Allow numeric tokens containing 'e' that aren't exponents be passed to proc macros

[ogoffart]

I tried this code:

// This could also be a complex proc macro
macro_rules! print_token {
    ($x:tt) => { println!("{}", stringify!($x)) }
}

fn main() {
    print_token!(123aefg123);  // ok, unknown suffix
    print_token!(123e123); // ok: float literal
    
    print_token!(123ef3); // error:  expected at least one digit in exponent
}

I expected to see this happen: It should compile in print the token verbatim. If unkown/invalid suffix are OK, so should invalid float, because the macro DSL my use it for something else

Instead, this happened: Compilation error :

error: expected at least one digit in exponent
  --> src/main.rs:11:18
   |
11 |     print_token!(123ef3); // error:  expected at least one digit in exponent
   |                  ^^^^^^

(tested with Rust 1.69 as well as current nightly, this always has been a problem)

Context

This was discussed with @nnethercote and others last week and I wanted to make sure there is a track of this.

The usecase is so that Slint's color literal work in the slint! maco DSL.
Currently one can do color: #0e9 but not color:#0ea in the slint! macro. And Makepad DSL has to workaround the same issue.

[makepaddev]

Chiming in that this is also absolutely a problem for our project (github.com/makepad/makepad) and would really appreciate a fix!. All proc-macro embedded UI systems for Rust need this.

[chenyukang]

This error comes out at next_token, which is too early.
So it's a general issue, here is another case I read from code:

macro_rules! print_token {
    ($x:tt) => { println!("{}", stringify!($x)) }
}

fn main() {
    print_token!("hello"_);
}

Output:

error: underscore literal suffix is not allowed
  --> ./p/mac.rs:11:25
   |
11 |     print_token!("hello"_);
   |                         ^

Seems not easy to fix, we need to change to report this kind of error after macro expanding.

[nnethercote]

I have a partial implementation that works for most literals involving an 'e', including CSS colours of the form #xxyyzz. I need to extend it a bit more to properly distinguish cases like 1e+_______3 (a valid float literal) and 1e+_______a (not a valid float literal, and currently rejected by the lexer).

From an implementation point of view it shouldn't be hard, but the change will need sign-off from the lang team. The relevant section of the Reference will need changing; in particular the Reserved forms similar to literals section as well as the SUFFIX_NO_E nonterminal in the grammar. Working through all this will take some time.

@chenyukang "hello"_ is a different issue to the one facing @ogoffart and @makepaddev. _ is not accepted as a suffix because a suffix must be a valid identifier or keyword, and _ is not. Let's keep this issue about numeric tokens with 'e' in them.

[nnethercote]

#111628 has an draft implementation. Zulip discussion is here.