Add portable lenient UTF-8/16 conversion routines #70
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I decided to split the implementation of UTF conversion utilities and UTF8-CPP was chosen after investigating various variants from this article. Also, despite the ODBC spec only officially supports UCS-2 (BMP only, no surrogates), the conversion functions expect UTF-16 because client applications may not obey the UCS-2-only requirement. edit: removed open questions, should be all covered in the cleaned-up updated version. |
staticlibs
force-pushed
the
widechar_conversion_routines
branch
3 times, most recently
from
0852c35 to
6a9f756
Compare
This change add support for lenient UTF-16 to/from UTF-8 conversion routines. It is implemented using UTF8-CPP library. The intention is to use these routines in the implementation of duckdb#69. Testing: tests added covering conversion routines and addtional helper functions.
staticlibs
force-pushed
the
widechar_conversion_routines
branch
from
6a9f756 to
01bb3f4
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change add support for lenient UTF-16 to/from UTF-8 conversion routines. It is implemented using UTF8-CPP library. The intention is to use these routines in the implementation of #69.
Testing: tests added covering conversion routines and addtional helper functions not provided by UTF8-CPP.