Move non-ascii-idents content from unstable book to reference.

[crlf0710]

non-ascii-idents feature is now in fcp. Moving its unstable book contents to reference.

[ehuss]

Thanks for the PR!

Would you be willing to help updating with the details of various restrictions and behaviors? Or at least help identify the list of them? The things I'm aware of:

• Explicitly specify the Unicode version.
• Does not allow non-ascii idents:
  • File-loaded modules
  • no_mangle items
  • extern crate
  • paths referencing external crates
• The RFC mentioned rejecting non-ascii extern mod items, was that removed?
• How does normalization work? If it is the same as specified in UAX #31, maybe it can just link to that? If not, then it should probably be spelled out clearly. Also, when does the normalization happen (particularly in relation to macros)?

[crlf0710]

Thanks for the comment!

1. Unicode version. Following Rust's overall Unicode strategy, the strategy here is to update to the newest release of new versions of Unicode on a regular basis. We rely on Unicode's backward compatibility guarantee to make sure existing code doesn't broke. Notability, the confusable detection lints related to this feature doesn't have this guarentee (the confusable set is expected to grow over time), which is why we advice that these lints' level not set to forbid

From the implementation side, unicode-related data for this feature is scattered within in unicode-xid, unicode-normalization and unicode-security. Currently all of them are using Unicode 13.0, and all these crates (and a few others, including libcore) needs an upgrade when new Unicode version is released.

2. Nothing to add here, but "paths referencing external crates" cannot use non-ascii idents only in the crate name part.

3. This was not mentioned during the implementation. I think this might be an oversight. I'll create a rustc issue for it. (Non-ASCII identifiers should be restricted within extern blocks. rust#83923)

4. Normalization is according to the RFC, using NFC. There is not exactly the same with UAX #31, since that recommends normalizing to NFKC instead. The reasons are described within the RFC text.

Time when the normalization happens:

1. For casual lexing, whenever before a symbol is created, the symbol string is normalized to nfc form first.
2. For proc-macros, whenever the proc-macro-server crate creates an identifier with Ident::new, the symbol is normalized and re-interned.

[Manishearth]

I think this should be good to go when rust-lang/rust#83799 lands. It would be nice to mention that "in some contexts non-ascii identifiers are not allowed, like extern functions"

[Manishearth]

Merging as is since this is strictly an improvement, but we should probably work on adding more about this.