Add ICU4X data generation from worldwide's CLDR #410
+5,720
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This adds a Rust implementation for ICU4X data generation from worldwide's patched CLDR data. The implementation includes: - Two Rust crates: * worldwide-icu4x-datagen: Build tool for generating data blobs * worldwide-icu4x-data: Published crate with embedded blob - Ruby integration: * rake icu4x:all task for full pipeline * JSON exporter to convert YAML to CLDR JSON format * Parity tests comparing Ruby/Rust outputs - CI/CD workflows: * ci-rust.yml: Build, test, and validate on PRs * rust-release.yml: Automated publishing to crates.io Current limitation: The datagen currently uses ICU4X's test data rather than worldwide's custom patches due to hour cycle format issues in the CLDR JSON export. This needs to be resolved before the full pipeline works with custom patches.
- rake icu4x:datagen: Run datagen without convert step - rake icu4x:datagen:debug: Run with full backtrace for debugging Useful for debugging hour cycle issues with existing CLDR JSON.
- Set RAYON_NUM_THREADS=1 for single-threaded execution - Set RUST_LOG=debug for verbose logging - Add --verbose flag to cargo - Print debug environment variables Makes it easier to trace which locale causes the hour cycle error.
Change exception: false to exception: true so the rake task exits immediately when datagen panics.
Add panic = 'abort' to both dev and release profiles so the datagen binary exits immediately on panic instead of unwinding. This prevents multiple threads from continuing to panic and makes debugging easier.
Remove panic = 'abort' as requested. The default panic behavior (unwinding) is sufficient since: - RAYON_NUM_THREADS=1 ensures single-threaded execution - exception: true in rake task ensures task fails on non-zero exit - Rust already exits with code 101 on panic
In debug mode: - Set custom panic hook that calls std::process::abort() - Run without --release flag to use debug build - This prevents Rayon from catching and recovering from panics Now 'bundle exec rake icu4x:datagen:debug' will stop on the very first locale that hits the hour cycle assertion error.
Parse debug output to track which locales are being processed and automatically identify which locale caused the panic. On failure, the task now prints: - Last 5 locale files read before panic - The likely problematic locale - Suggested next step (which YAML file to inspect) Uses Open3.popen3 to capture and parse stderr while forwarding output to the console in real-time.
Load from vendor/cldr-json/ instead of ICU4X built-in data: - Use SourceData::new().with_cldr_json() to load custom data - Add validation that vendor/cldr-json/ exists before running - Update logging to show we're using worldwide patched data - Remove TODO comments - custom CLDR loading is now implemented This allows the datagen to use Shopify's patched CLDR data instead of ICU4X's built-in data which has hour cycle bugs.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds Rust infrastructure to generate ICU4X data blobs from worldwide's patched CLDR data. The goal is to enable high-performance i18n operations in native code while ensuring parity with Ruby.
What's included
Currently supported
Not yet supported
Decimal formatting is blocked because ICU4X requires
decimalFormats-numberSystem-latnwith format patterns (e.g.,#,##0.###) in addition to symbols. Worldwide's YAML only has symbols. The parity tests are in place but skipped.Usage