Tools: add a script to download Unicode data files, and rename grapheme_break_property_data_gen.py to unicode_properties_data_gen.py
#4435
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cpplearner
force-pushed
the
download-unicode-data
branch
from
4a5b648
to
733d66c
Compare
cpplearner
force-pushed
the
download-unicode-data
branch
from
733d66c
to
d11ad04
Compare
StephanTLavavej
approved these changes
Mar 5, 2024
StephanTLavavej
approved these changes
Mar 6, 2024
|
I'm mirroring this to the MSVC-internal repo - please notify me if any further changes are pushed. |
|
Thanks for simplifying this process - I've wanted a download script for a while! 🪄 😻 🎉 |
tahonermann
reviewed
Mar 9, 2024
Comment on lines
+7
to
+14
| Unicode_data_files = { | ||
| "DerivedCoreProperties.txt": "https://www.unicode.org/Public/UCD/latest/ucd/DerivedCoreProperties.txt", | ||
| "DerivedGeneralCategory.txt": "https://www.unicode.org/Public/UCD/latest/ucd/extracted/DerivedGeneralCategory.txt", | ||
| "EastAsianWidth.txt": "https://www.unicode.org/Public/UCD/latest/ucd/EastAsianWidth.txt", | ||
| "GraphemeBreakProperty.txt": "https://www.unicode.org/Public/UCD/latest/ucd/auxiliary/GraphemeBreakProperty.txt", | ||
| "GraphemeBreakTest.txt": "https://www.unicode.org/Public/UCD/latest/ucd/auxiliary/GraphemeBreakTest.txt", | ||
| "emoji-data.txt": "https://www.unicode.org/Public/UCD/latest/ucd/emoji/emoji-data.txt", | ||
| } |
There was a problem hiding this comment.
Is it desirable that these pull data from the latest available UCD files? That complicates rerunning the scripts with the intention to match what might have been downloaded previously. If it is intentional, perhaps rename the script to download_latest_unicode_data_files.py. Otherwise, the files specific to Unicode 15.1.0 (for example) are available at https://unicode.org/Public/15.1.0/ucd. Another alternative would be to allow a base Unicode version URL to be supplied on the command line.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
These are attempts to make it easier to regenerate source files from the Unicode data.
SG16 recommends specifying Unicode 15.1.0 as the minimum Unicode version for C++23 (as a DR) and C++26. See cplusplus/papers#1736 and https://github.com/sg16-unicode/sg16-meetings.
Currently,
__msvc_format_ucd_tables.hppis based on Unicode 15.0.0. In order to support 15.1.0, this file will need to be regenerated.(Other changes are also needed. For example,
_Grapheme_break_property_iteratormight need to account for the changes in UAX 29. I haven't taken a close look, though.)