uncommon_codepoints is only checked post-NFC

[Jules-Bertholet]

Code

#![forbid(uncommon_codepoints)]
pub const L·L: u32 = 7;

Current output

(compiles successfully)

Desired output

error: identifier contains an uncommon Unicode codepoint: '·'
 --> src/lib.rs:2:11
  |
2 | pub const L·L: u32 = 7;
  |           ^^^
  |

Rationale and extra context

The · in the above code snippet is U+0387 GREEK ANO TELEIA, which has an Identifier_Status of Restricted and should therefore trigger the uncommon_codepoints lint. However, U+0387 has an NFC decomposition to U+00B7 ( · ) MIDDLE DOT, which has an Identifier_Status of Allowed, and is therefore not flagged by the lint. Because the compiler applies NFC normalization to identifiers before checking uncommon_codepoints, the lint incorrectly fails to fire in this case.

Rust Version

rustc 1.75.0 (82e1608df 2023-12-21)
binary: rustc
commit-hash: 82e1608dfa6e0b5569232559e3d385fea5a93112
commit-date: 2023-12-21
host: x86_64-unknown-linux-gnu
release: 1.75.0
LLVM version: 17.0.6

@rustbot label A-unicode

[Jules-Bertholet]

Alternatively to or along with fixing this, perhaps rustfmt should NFC-normalize identifiers?

[workingjubilee]

Should we lint against non-NFC-normalized idents?

[workingjubilee]

Opened rust-lang/rustfmt#6058

[workingjubilee]

A program that makes the difference visible to human eyes: https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=8a408f3aba0d4a3c99a1fbf85f9c4473

fn main() {
    for id in ["·", stringify!(L·L), "L·L", stringify!(_·), "_·"] {
        println!("printing {len} bytes of: {id}", len = id.len());
        for c in id.chars() {
            let mut bytes = [0; 4];
            c.encode_utf8(&mut bytes);
            for byte in bytes.iter().take(c.len_utf8()) {
                print!("{:x} ", byte);
            }
        }
        println!("");
    }
}