Unicode Normalization of Identifiers

[tamaroning]

Related to #2287

All identifiers must be normalized to the NFC form.
Normalization is used to treat equally strings which consists of different codepoints but practiaclly the same.
It is useful when a program is linked to other Rust libraries,

NFC Normalization takes two steps:

1. Some kinds of characters are decomposed to one or multiple chatacters. (This is called NFD form)
2. Then, some kinds of chatacters are composed to a single character. (This is called NFC form)

For example,
https://unicode.org/reports/tr15/images/UAX15-NormFig4.jpg

Implementation

Unicode data file parser

• written in python
  • Add header file generator for Unicode normalization and alphanumeric check #2425
• add type annotation for mypy
  • Add type annotation to make-rust-unicode-data.py #2529

Decomposition

• recursive decomposition with Decompostion_Mapping
  • Add header file generator for Unicode normalization and alphanumeric check #2425
• algorithmic decomposition of Hangul
• sort characters with Canonical_Combining_Class
  • Add header file generator for Unicode normalization and alphanumeric check #2425
• algorithmic decomposition of Hangul
  • Normalize Hangul #2467

Composition

• recompose a starter with its following chartacters if composable
  • Add header file generator for Unicode normalization and alphanumeric check #2425
• do not recompose characters with the Composition_Exclusion property
  • Add header file generator for Unicode normalization and alphanumeric check #2425
• algorithmic composition of Hangul
  • Normalize Hangul #2467

Optimization

• quick check
• use trie for composition mapping table
  • but the current implementation is not so slow

Tests

many testcases are needed!

Ref

• https://doc.rust-lang.org/reference/identifiers.html#normalization
• https://unicode.org/reports/tr15/
• Reference implemetation: https://www.w3.org/International/charlint/