Docs for utf8 decoding #2979
Docs for utf8 decoding #2979
Conversation
|
CC @CremboC |
| * Note that the output stream is ''not'' a singleton stream but rather a stream | ||
| * of strings where each string is the result of UTF8 decoding a chunk of the | ||
| * underlying byte stream. | ||
| */ |
There was a problem hiding this comment.
Oh right, I can see how this was confusing 😅 thanks!
If we really get nitty, IIUC it's not technically not one-string-per-Chunk, since some multi-byte characters could be split across Chunks. Not sure if there's a good way to say that though, or if it really matters.
There was a problem hiding this comment.
Oh haha ouch! I suspect attempting to explain that would cause more confusion rather than less but I'm very happy if someone can suggest a simple explanation for it!
There was a problem hiding this comment.
It is even more complicated, since a chunk from the input could just be the middle bytes of a multi-byte character, so not a single character would be built off that chunk.
The result is a stream of strings. Every chunk in the result contains exactly one string, and each string carries all characters that could be fully decoded from the input.
But I think it may be easier to just say
For the most part, each string in the output is the result of decoding a chunk of bytes from the input; however, this may not be accurate when the bytes of a multi-byte character are split amongst one or more input chunks.
Clarify a potentially confusing aspect of the decoding API