Recursive macro bypasses recursion limit, causes memory leak

[BigWingBeat]

The following code causes rustc to get stuck on compiling while memory usage skyrockets, until it gets killed by the OOM killer.

macro_rules! recursive {
    ($($t:tt)*) => {
        recursive!($($t)* asdasd $($t)*);
    };
}

fn main() {
    recursive!();
}

The most important part of this is the asdasd text inbetween the two expansions. The memory leak only happens when some non-whitespace text is present there. I've tested several variations of text in that position and they all caused the issue:

• recursive!($($t)* asdasd $($t)*);
• recursive!($($t)* ; $($t)*);
• recursive!($($t)* | $($t)*);
• recursive!($($t)* , $($t)*);

etc.

That is to say, when that line of code only has whitespace inbetween the two expansions: recursive!($($t)* $($t)*); the issue does not occur, and the compiler instantly terminates with a recursion limit error as it should:

Output with only whitespace

   Compiling macro-test v0.1.0 (/home/ts/dev/rust/macro-test)
error: recursion limit reached while expanding `recursive!`
 --> src/main.rs:3:9
  |
3 |         recursive!($($t)* $($t)*);
  |         ^^^^^^^^^^^^^^^^^^^^^^^^^
...
8 |     recursive!();
  |     ------------ in this macro invocation
  |
  = help: consider increasing the recursion limit by adding a `#![recursion_limit = "256"]` attribute to your crate (`macro_test`)
  = note: this error originates in the macro `recursive` (in Nightly builds, run with -Z macro-backtrace for more info)

error: could not compile `macro-test` due to previous error

Meta

This issue occurs both on stable and nightly.

rustc --version --verbose:

rustc 1.66.0 (69f9c33d7 2022-12-12)
binary: rustc
commit-hash: 69f9c33d71c871fc16ac445211281c6e7a340943
commit-date: 2022-12-12
host: x86_64-unknown-linux-gnu
release: 1.66.0
LLVM version: 15.0.2

rustc 1.68.0-nightly (9c07efe84 2022-12-16)
binary: rustc
commit-hash: 9c07efe84f28a44f3044237696acc295aa407ee5
commit-date: 2022-12-16
host: x86_64-unknown-linux-gnu
release: 1.68.0-nightly
LLVM version: 15.0.6

Backtrace

When run via cargo build (After stalling for ~30s):

ts@pop-os:~/dev/rust/macro-test$ cargo build
   Compiling macro-test v0.1.0 (/home/ts/dev/rust/macro-test)
error: could not compile `macro-test`

Caused by:
  process didn't exit successfully: `rustc --crate-name macro_test --edition=2021 src/main.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C opt-level=1 -C panic=abort -C embed-bitcode=no -C debuginfo=2 -C debug-assertions=on -C metadata=d503706fc921fb16 -C extra-filename=-d503706fc921fb16 --out-dir /home/ts/.cache/cargo/target/debug/deps -C linker=clang -C incremental=/home/ts/.cache/cargo/target/debug/incremental -L dependency=/home/ts/.cache/cargo/target/debug/deps -C link-arg=-fuse-ld=lld -C target-cpu=native` (signal: 9, SIGKILL: kill)

When run via rustc src/main.rs (After stalling for ~30s):

ts@pop-os:~/dev/rust/macro-test$ rustc -v src/main.rs
Killed

Adding RUST_BACKTRACE=1 to either of the above commands does not change the output at all.

[Noratrieb]

Our default recursion limit for macros is over 100, and 2^100 is a really, really, really big number already. I guess we need some sort of limit on the length of the token stream to guard against these exponential macros.

[jyn514]

cc #95698, https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/rustc_expand.20keep.20into.20count.20expansion.20limit.20also.20the, #103029. I think we can close this as a duplicate of #95698.