The output char buffer is too small to contain the decoded characters...

[n9]

I am getting the following the exception from JsonRpc.Completion.

The output char buffer is too small to contain the decoded characters, encoding 'Unicode (UTF-8)' fallback 'System.Text.DecoderReplacementFallback'. (Parameter 'chars')

Stack trace is:

   at System.Text.Encoding.ThrowCharsOverflow()
   at System.Text.Encoding.ThrowCharsOverflow(DecoderNLS decoder, Boolean nothingDecoded)
   at System.Text.Encoding.GetCharsWithFallback(ReadOnlySpan`1 bytes, Int32 originalBytesLength, Span`1 chars, Int32 originalCharsLength, DecoderNLS decoder)
   at System.Text.UTF8Encoding.GetCharsWithFallback(ReadOnlySpan`1 bytes, Int32 originalBytesLength, Span`1 chars, Int32 originalCharsLength, DecoderNLS decoder)
   at System.Text.Encoding.GetCharsWithFallback(Byte* pOriginalBytes, Int32 originalByteCount, Char* pOriginalChars, Int32 originalCharCount, Int32 bytesConsumedSoFar, Int32 charsWrittenSoFar, DecoderNLS decoder)
   at System.Text.Encoding.GetChars(Byte* pBytes, Int32 byteCount, Char* pChars, Int32 charCount, DecoderNLS decoder)
   at System.Text.DecoderNLS.Convert(Byte* bytes, Int32 byteCount, Char* chars, Int32 charCount, Boolean flush, Int32& bytesUsed, Int32& charsUsed, Boolean& completed)
   at System.Text.DecoderNLS.Convert(Byte[] bytes, Int32 byteIndex, Int32 byteCount, Char[] chars, Int32 charIndex, Int32 charCount, Boolean flush, Int32& bytesUsed, Int32& charsUsed, Boolean& completed)
   at Nerdbank.Streams.SequenceTextReader.DecodeChars()
   at Nerdbank.Streams.SequenceTextReader.DecodeCharsIfNecessary()
   at Nerdbank.Streams.SequenceTextReader.Read(Char[] buffer, Int32 index, Int32 count)
   at Newtonsoft.Json.JsonTextReader.ReadData(Boolean append, Int32 charsRequired)
   at Newtonsoft.Json.JsonTextReader.ReadStringIntoBuffer(Char quote)
   at Newtonsoft.Json.JsonTextReader.ParseProperty()
   at Newtonsoft.Json.Linq.JContainer.ReadContentFrom(JsonReader r, JsonLoadSettings settings)
   at Newtonsoft.Json.Linq.JContainer.ReadTokenFrom(JsonReader reader, JsonLoadSettings options)
   at Newtonsoft.Json.Linq.JObject.Load(JsonReader reader, JsonLoadSettings settings)
   at Newtonsoft.Json.Linq.JToken.ReadFrom(JsonReader reader, JsonLoadSettings settings)
   at Newtonsoft.Json.Linq.JToken.ReadFrom(JsonReader reader)
   at StreamJsonRpc.JsonMessageFormatter.ReadJToken(ReadOnlySequence`1 contentBuffer, Encoding encoding)
   at StreamJsonRpc.JsonMessageFormatter.Deserialize(ReadOnlySequence`1 contentBuffer, Encoding encoding)
   at StreamJsonRpc.JsonMessageFormatter.Deserialize(ReadOnlySequence`1 contentBuffer)
   at StreamJsonRpc.WebSocketMessageHandler.<ReadCoreAsync>d__10.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at StreamJsonRpc.MessageHandlerBase.<ReadAsync>d__22.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at StreamJsonRpc.JsonRpc.<ReadAndHandleRequestsAsync>d__132.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

In addition to this exception, I am getting another exception from the method call:

The JSON-RPC connection with the remote party was lost before the request could complete.

with stack trace:

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at StreamJsonRpc.JsonRpc.<InvokeCoreAsync>d__125.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at StreamJsonRpc.JsonRpc.<InvokeCoreAsync>d__115`1.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()

How to find out which exception is cause and which is effect?
Does RPC call fail because of the first exception: buffer issue?
Or does RPC call fail because of the second exception: remote party was lost?

[AArnott]

I expect the first exception you mentioned happens first. Since decoding a message is a protocol critical error, we disconnect, leading to the second one.

The relevant code in the first exception is here:
https://github.com/AArnott/Nerdbank.Streams/blob/e86aad233584adfbbe135a9154c24f59e29f9045/src/Nerdbank.Streams/SequenceTextReader.cs#L285

Do you have a solid repro? I'd like to investigate and fix this.

[AArnott]

What runtime and version are you on (e.g. .NET Framework or .NET Core x.y)?

The Decoder.Convert method is not supposed to throw this exception, per this documentation. So this looks like it might be a runtime bug.

[n9]

The exception that happens first is the from rpc call remote party was lost. Then "JsonRpc.Completion" fails with buffer is too small.

Both sides use .NET Core 3.1.

[AArnott]

Thanks. I'm still pretty sure the decoding failure causes the connection loss, even if you observe the exceptions in the opposite order. But you can see in the Disconnected event that there's a reason property that should tell you why the disconnection happened for sure.

I have a repro and filed a bug as linked above. I'll keep this issue open to track updating to the nerdbank.streams library that has the fix.

[n9]

Yes. I have added the Disconnected handler:

{StreamJsonRpc.JsonRpcDisconnectedEventArgs}
    Description: "Reading JSON RPC from the stream failed with ArgumentException: The output char buffer is too small to contain the decoded characters, encoding 'Unicode (UTF-8)' fallback 'System.Text.DecoderReplacementFallback'. (Parameter 'chars')"
    Exception: {"The output char buffer is too small to contain the decoded characters, encoding 'Unicode (UTF-8)' fallback 'System.Text.DecoderReplacementFallback'. (Parameter 'chars')"}
    LastMessage: null
    Reason: StreamError

Now the order is following:

1. Disconnected handler (buffer is too small)
2. RPC call exception (remote party was lost)
3. Completion exception (buffer is too small)

I can confirm that my case is using surrogate pairs.

[AArnott]

@n9 For now you can workaround the problem by installing the Nerdbank.Streams package directly in your project, with version 2.4.67 or above.

[n9]

@AArnott Thank you. It is no longer failing.