Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix UnicodeDecodeError for Chinese #75

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

Brightcells
Copy link

ipdb> value
'\xe8\xa5\xbf\xe6\xb1\x89\xe6\x96\x87\xe5\xad\xa6\xe6\x88\x90\xe5\xb0\xb1\xe4\xb8\xad\xef\xbc\x8c\xe6\x9c\x80\xe4\xb8\xba\xe7\xaa\x81\xe5\x87\xba\xe7\x9a\x84\xe6\x98\xaf\xef\xbc\x9f'
ipdb> unicode(value)
*** UnicodeDecodeError: 'ascii' codec can't decode byte 0xe8 in position 0: ordinal not in range(128)
ipdb> cc.Convert2Unicode(value)

u'\u897f\u6c49\u6587\u5b66\u6210\u5c31\u4e2d\uff0c\u6700\u4e3a\u7a81\u51fa\u7684\u662f\uff1f'
ipdb> print cc.Convert2Unicode(value)

西汉文学成就中最为突出的是ipdb>

@Parkayun
Copy link
Member

@Brightcells
Hi. Can I know why use CodeConvert instead of to write function?
I want this package has minimum dependency.
And test failed.

@Brightcells
Copy link
Author

Chinese character may be utf8 or gbk.
I does not to test whether Chinese character in .bson file can be gbk or not.
Instead I use CodeConvert to realize convert to unicode.

@Brightcells
Copy link
Author

Fixed travis tests

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants