Port fix: Fix JSPB binary utf8 decoding and validate by default (breaking change) #191
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lukesandberg
approved these changes
Mar 2, 2024
| @@ -354,7 +354,7 @@ describe('binaryDecoderTest', () => { | |||
|
|
|||
| const decoder = jspb.BinaryDecoder.alloc(encoder.end()); | |||
|
|
|||
| expect(decoder.readString(len)).toEqual(long_string); | |||
| expect(decoder.readString(len, true)).toEqual(long_string); | |||
There was a problem hiding this comment.
Suggested change
| expect(decoder.readString(len, true)).toEqual(long_string); | |
| expect(decoder.readString(len, /* enforceUtf8= */ true)).toEqual(long_string); |
dibenede
changed the title
dibenede
changed the title
|
What's the status of this? Is there a reason it hasn't been merged? |
This will be going in shortly. I wanted to cut a release beforehand, which is now done. However, that was slow to get out the door due to release infrastructure issues. |
Our prior behavior was extremely undefined when confronted with errors, it would read out of bounds, accept overlong encodings, skip over out of range bytes, compose out of range codepoints. The new implementation always detects and handles errors consistently by either throwing or using replacement characters (� aka \uFFFD) This also adds support for aligning with the proto3 spec to the code generator which requires that parsing fail for proto3 messages with invalid utf8 payloads for string fields. For now, actual failing is disabled via the goog.define jspb.binary.ENFORCE_UTF8 which is set to NEVER. A future change will flip this to DEFAULT.
Co-authored-by: Luke Sandberg <lukesandberg@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Our prior behavior was extremely undefined when confronted with errors, it would read out of bounds, accept overlong encodings, skip over out of range bytes, compose out of range codepoints. The new implementation always detects and handles errors consistently by either throwing or using replacement characters (� aka \uFFFD)
This also adds support for aligning with the proto3 spec to the code generator which requires that parsing fail for proto3 messages with invalid utf8 payloads for string fields. For now, actual failing is disabled via the goog.define jspb.binary.ENFORCE_UTF8 which is set to NEVER. A future change will flip this to DEFAULT.