Avoid cutting off utf8 encoding halfway when truncating text values #148
Labels
Milestone
Comments
karimbahgat
added a commit
that referenced
this issue
Sep 8, 2018
- Fixes issue in Py3 when converting text characters to byte strings, but in Py3 converts to unicode instead, because uses the Py2 specific str() function, instead of the version neutral b(). When the text contains non-ascii 2-byte unicode values this results in truncating the unicode length instead of the byte length, and thus results in incorrectly padded byte lengths and data values ending up in the wrong field/column. See #157, and also #148. - Also bump to next version.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Currently encodes any unicode to bytes (using whichever encoding), and truncates afterwards to fit within the specified byte size of the field. I guess UTF8 and any encoding that allows characters that spans more than a single byte would stand at risk of cutting and invalidating such characters if they occur at the end of the text, spanning across the truncating limit. See #125.
The text was updated successfully, but these errors were encountered: