Add support for repetition to `proc_macro::quote` #141608

moatom · 2025-05-26T17:47:55Z

Progress toward: #140238

moatom · 2025-05-31T08:03:17Z

library/proc_macro/src/lib.rs

@@ -1613,3 +1614,202 @@ pub mod tracked_path {
        crate::bridge::client::FreeFunctions::track_path(path);
    }
 }
+
+#[doc(hidden)]
+#[unstable(feature = "proc_macro_quote", issue = "54722")]


I am not sure whether these annotations are appropriate. (I added them just based on my speculation.)

In addition,

It's probably easiest to just copy quote's logic here, which uses an extension trait to facilitate this.

do we need to take care of its license?

I am not sure whether these annotations are appropriate. (I added them just based on my speculation.)

You can always try removing them and see if it complains - but in general, all crate-public items need a stability gate. Try to make as little as possible actually pub of course, but that's difficult because this needs a lot of helpers to be public.

This all doesn't need to be in lib.rs though, could you move the additions here to the quote module? And then reexport only what is needed.

It's probably easiest to just copy quote's logic here, which uses an extension trait to facilitate this.

do we need to take care of its license?

quote is MIT AND Apache-2.0, which is the same as rust-lang/rust so there is no problem here.

This all doesn't need to be in lib.rs though, could you move the additions here to the quote module? And then reexport only what is needed.

DONE.
FYI, I used #[unstable(feature = "proc_macro_quote", issue = "54722")] instead of something like #[unstable(feature = "proc_macro_quote_span", issue = "140238")], because this PR modifies the quote function, which is already part of the "proc_macro_quote" feature.

moatom · 2025-05-31T08:08:18Z

library/proc_macro/src/lib.rs

+#[unstable(feature = "proc_macro_quote", issue = "54722")]
+pub mod ext {
+    use core::slice;
+    use std::collections::btree_set::{self, BTreeSet};


an alternative for alloc::collections

moatom · 2025-05-31T08:12:55Z

library/proc_macro/src/quote.rs

+        }
+
+        self.tokens[self.pos] = self.iter.next();
+        let token_opt = self.tokens[self.pos].clone();


This helper struct may require significant refactoring. (Especially, the use of .clone() might be avoidable.)
Do you have any comments or suggestions?

Might be worth using just a Peekable. I think it would be better to use it at the expense of some extra parsing code, than using a custom lookahead iterator

@ora-0
I think Peekable only supports single-element lookahead.

However, placing a simplified and extended version of Peekable here might be a better approach than defining the completely original lookahead iterator from scratch. 🤔

I'm not sure if a lookahead is necessary. Would this work? I can't be completely sure since I haven't tested this code.

let mut iter = stream.into_iter(); // ... let mut sep_opt: Option<TokenTree> = None; if let next @ Some(TokenTree::Punct(token_1)) = iter.next() { if token_1.as_char() != '*' { sep_opt = next; if !matches!(iter.next(), Some(TokenTree::Punct(token_2)) if token_2 == "*") { panic!("`$(...)` must be followed by `*` in `quote!`"); } } }

Since we are panicking at the wildcard we could just use .next()s. It is a bit less declarative but it avoids the complexity of another data structure.

Nvm, turns out it is necessary, because the separator itself may end up being a star. But I think we only need one lookahead. The code ends up being pretty similar to the current:

let mut iter = stream.into_iter().peekable(); // ... let sep_opt: Option<TokenTree> = match (iter.next(), iter.peek()) { (Some(TokenTree::Punct(sep)), Some(&TokenTree::Punct(star))) if sep.spacing() == Spacing::Joint && star.as_char() == '*' => { iter.next(); Some(TokenTree::Punct(sep)) } (Some(TokenTree::Punct(star)), _) if star.as_char() == '*' => None, _ => panic!("`$(...)` must be followed by `*` in `quote!`"), };

But I think we only need one lookahead.

Thank you for the great point! We must consume at least one * here.

moatom · 2025-05-31T08:16:27Z

library/proc_macro/src/quote.rs

@@ -71,10 +160,97 @@ pub fn quote(stream: TokenStream) -> TokenStream {
    let mut after_dollar = false;

    let mut tokens = crate::TokenStream::new();
-    for tree in stream {
+    let mut iter = LookaheadIter::new(stream);
+    while let Some(tree) = iter.next() {
        if after_dollar {


after_dollar can be combined with LookaheadIter, if you prefer.

moatom · 2025-05-31T08:24:58Z

tests/ui/proc-macro/quote/auxiliary/basic.rs

+    assert_eq!("X, X, X, X,", quote!($($primes,)*).to_string());
+
+    assert_eq!("X, X, X, X", quote!($($primes),*).to_string());


I adjusted the expected spacing around SEP in the original test code. Is it appropriate?
Cf. https://github.com/dtolnay/quote/blob/62fd385a800f7398ab416c00100664479261a86e/tests/test.rs#L84

proc_macro and proc_macro2 unquote whitespace slightly differently, is this what you are referring to? If so, I don't think there is any problem.

tests/ui/proc-macro/quote/does-not-have-iter-interpolated-dup.rs

rustbot · 2025-06-01T00:48:18Z

r? @petrochenkov

rustbot has assigned @petrochenkov.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

moatom · 2025-06-01T00:53:34Z

@tgross35 @dtolnay @ora-0
Hi! Although it still needs some refactoring, it passes all the updated tests.
I'd be grateful for your review and feedback.

moatom · 2025-06-01T00:53:43Z

r? @tgross35

tgross35 · 2025-06-01T04:39:31Z

I’ll leave a review but David knows this area much better, so

r? dtolnay

tgross35 · 2025-06-13T18:02:48Z

tests/ui/proc-macro/quote/auxiliary/basic.rs

+    assert_eq!("X, X, X, X,", quote!($($primes,)*).to_string());
+
+    assert_eq!("X, X, X, X", quote!($($primes),*).to_string());


proc_macro and proc_macro2 unquote whitespace slightly differently, is this what you are referring to? If so, I don't think there is any problem.

tests/ui/proc-macro/quote/does-not-have-iter-interpolated-dup.rs

tgross35 · 2025-06-13T18:10:40Z

library/proc_macro/src/quote.rs

        if after_dollar {
            after_dollar = false;
            match tree {
+                TokenTree::Group(inner) => {


Could you add comments in this section about what is happening?

I added them, but

https://github.com/rust-lang/rust/pull/141608/files#diff-6ebaa6be050b7cea1d542579c11cde3cba0942a3417c53298001f9db02a8d06dR310-R311

// Append setup code for a while, where recursively quoted CONTENTS
// and SEP_OPT are repeatedly processed, to REP_EXPANDED.

the word choice of "setup code" may be ambiguous.
Moreover, it would be better to carefully distinguish the terms "expanded", "processed", and "quoted".

tgross35 · 2025-06-13T18:11:06Z

library/proc_macro/src/quote.rs

@@ -155,6 +276,30 @@ pub fn quote(stream: TokenStream) -> TokenStream {
    }
 }

+fn collect_meta_vars(stream: TokenStream) -> Vec<Ident> {


Could you add a doc comment?

tgross35 · 2025-06-13T18:14:23Z

You should also rebase when you update this, I think the CI failure was from something spurious.

moatom · 2025-06-16T18:32:23Z

@tgross35 @dtolnay
I've addressed all the points. Please review it at your convenience.

@rustbot rustbot added the A-LLVM label Jun 17, 2025

This label was added because I mistakenly pushed incorrectly rebased commits. Feel free to remove it if you'd like.

dtolnay · 2025-06-16T21:08:47Z

Thanks for working on this feature!

I see some parsing differences here compared to what the quote crate does. I think this PR is a good start and is fine to land in nightly, but not all of these differences seem like improvements to me. Example:

#![feature(proc_macro_quote)]

use proc_macro::TokenStream;

macro_rules! decl {
    ($($iter:tt)*) => {
        stringify!($($iter) << *)
    };
}

#[proc_macro]
pub fn repro(input: TokenStream) -> TokenStream {
    // macro_rules macro
    let tokens = decl!(a b c);
    eprintln!("{}", tokens);

    // quote crate
    let input2 = proc_macro2::TokenStream::from(input.clone());
    let iter2 = input2.into_iter();
    let tokens = quote::quote!(#(#iter2) << *);
    eprintln!("{}", tokens);

    // libproc_macro
    let iter = input.into_iter();
    let tokens = proc_macro::quote!($($iter) << *);
    eprintln!("{}", tokens);

    TokenStream::new()
}

Macro_rules macro: a << b << c
Quote crate: a << b << c
Libproc_macro:

error: proc macro panicked
  --> src/lib.rs:25:18
   |
25 |     let tokens = proc_macro::quote!($($iter) << *);
   |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = help: message: `$(...)` must be followed by `*` in `quote!`

Another example:

error[E0425]: cannot find value `j` in this scope
  --> src/lib.rs:25:47
   |
25 |     let tokens = proc_macro::quote!($$ j $($$ j $iter)*);
   |                                               ^ not found in this scope

The parsing logic will need some more scrutiny in followup PRs before the macro can be stabilized.

@bors r+

bors · 2025-06-16T21:08:49Z

📌 Commit 23e35c6 has been approved by dtolnay

It is now in the queue for this repository.

tgross35 · 2025-06-16T21:16:27Z

I see some parsing differences here compared to what the quote crate does. I think this PR is a good start and is fine to land in nightly, but not all of these differences seem like improvements to me. Example:

I removed the tag to close #140238 and added your comment as a todo item there so we don't lose this follow up.

Add support for repetition to `proc_macro::quote` Progress toward: rust-lang#140238

Rollup of 11 pull requests Successful merges: - #140809 (Reduce special casing for the panic runtime) - #141608 (Add support for repetition to `proc_macro::quote`) - #141864 (Handle win32 separator for cygwin paths) - #142216 (Miscellaneous RefCell cleanups) - #142517 (Windows: Use anonymous pipes in Command) - #142570 (Reject union default field values) - #142584 (Handle same-crate macro for borrowck semicolon suggestion) - #142585 (Update books) - #142586 (Fold unnecessary `visit_struct_field_def` in AstValidator) - #142595 (Revert overeager warning for misuse of `--print native-static-libs`) - #142598 (Set elf e_flags on ppc64 targets according to abi) r? `@ghost` `@rustbot` modify labels: rollup

Rollup of 14 pull requests Successful merges: - #141574 (impl `Default` for `array::IntoIter`) - #141608 (Add support for repetition to `proc_macro::quote`) - #142100 (rustdoc: make srcIndex no longer a global variable) - #142371 (avoid `&mut P<T>` in `visit_expr` etc methods) - #142517 (Windows: Use anonymous pipes in Command) - #142520 (alloc: less static mut + some cleanup) - #142588 (Generic ctx imprv) - #142605 (Don't unwrap in enzyme builds in case of missing llvm-config) - #142608 (Refresh module-level docs for `rustc_target::spec`) - #142618 (Lint about `console` calls in rustdoc JS) - #142620 (Remove a panicking branch in `BorrowedCursor::advance`) - #142631 (Dont suggest remove semi inside macro expansion for redundant semi lint) - #142632 (Update cargo) - #142635 (Temporarily add back -Zwasm-c-abi=spec) r? `@ghost` `@rustbot` modify labels: rollup

Rollup merge of #141608 - moatom:proc_macro-140238, r=dtolnay Add support for repetition to `proc_macro::quote` Progress toward: #140238

Rollup of 14 pull requests Successful merges: - rust-lang/rust#141574 (impl `Default` for `array::IntoIter`) - rust-lang/rust#141608 (Add support for repetition to `proc_macro::quote`) - rust-lang/rust#142100 (rustdoc: make srcIndex no longer a global variable) - rust-lang/rust#142371 (avoid `&mut P<T>` in `visit_expr` etc methods) - rust-lang/rust#142517 (Windows: Use anonymous pipes in Command) - rust-lang/rust#142520 (alloc: less static mut + some cleanup) - rust-lang/rust#142588 (Generic ctx imprv) - rust-lang/rust#142605 (Don't unwrap in enzyme builds in case of missing llvm-config) - rust-lang/rust#142608 (Refresh module-level docs for `rustc_target::spec`) - rust-lang/rust#142618 (Lint about `console` calls in rustdoc JS) - rust-lang/rust#142620 (Remove a panicking branch in `BorrowedCursor::advance`) - rust-lang/rust#142631 (Dont suggest remove semi inside macro expansion for redundant semi lint) - rust-lang/rust#142632 (Update cargo) - rust-lang/rust#142635 (Temporarily add back -Zwasm-c-abi=spec) r? `@ghost` `@rustbot` modify labels: rollup

rustbot added the T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. label May 26, 2025

This comment has been minimized.

Sign in to view

moatom changed the title ~~WIP: Add support for repetition to proc_macro::quote~~ Add support for repetition to proc_macro::quote May 31, 2025

moatom commented May 31, 2025

View reviewed changes

tests/ui/proc-macro/quote/does-not-have-iter-interpolated-dup.rs Show resolved Hide resolved

moatom marked this pull request as ready for review June 1, 2025 00:48

rustbot assigned petrochenkov Jun 1, 2025

rustbot added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jun 1, 2025

This comment has been minimized.

Sign in to view

rustbot assigned tgross35 and unassigned petrochenkov Jun 1, 2025

rustbot assigned dtolnay and unassigned tgross35 Jun 1, 2025

This comment has been minimized.

Sign in to view

tgross35 reviewed Jun 13, 2025

View reviewed changes

moatom force-pushed the proc_macro-140238 branch 3 times, most recently from dcca29e to e6cb946 Compare June 16, 2025 16:33

rustbot added the A-LLVM Area: Code generation parts specific to LLVM. Both correctness bugs and optimization-related issues. label Jun 16, 2025

Add support for repetition to proc_macro::quote

23e35c6

moatom force-pushed the proc_macro-140238 branch from e6cb946 to 23e35c6 Compare June 16, 2025 16:40

dtolnay added A-proc-macros Area: Procedural macros and removed A-LLVM Area: Code generation parts specific to LLVM. Both correctness bugs and optimization-related issues. labels Jun 16, 2025

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 16, 2025

tgross35 mentioned this pull request Jun 16, 2025

Add support for repetition to proc_macro::quote #140238

Open

jhpratt added a commit to jhpratt/rust that referenced this pull request Jun 17, 2025

Rollup merge of rust-lang#141608 - moatom:proc_macro-140238, r=dtolnay

e698e9c

Add support for repetition to `proc_macro::quote` Progress toward: rust-lang#140238

jhpratt mentioned this pull request Jun 17, 2025

Rollup of 11 pull requests #142604

Closed

jhpratt mentioned this pull request Jun 17, 2025

Rollup of 14 pull requests #142644

Merged

bors merged commit 17ab49a into rust-lang:master Jun 18, 2025
10 checks passed

rustbot added this to the 1.89.0 milestone Jun 18, 2025

rust-timer added a commit that referenced this pull request Jun 18, 2025

Unrolled build for #141608

a45f31e

Rollup merge of #141608 - moatom:proc_macro-140238, r=dtolnay Add support for repetition to `proc_macro::quote` Progress toward: #140238

		assert_eq!("X, X, X, X,", quote!($($primes,)*).to_string());

		assert_eq!("X, X, X, X", quote!($($primes),*).to_string());

Add support for repetition to proc_macro::quote #141608

Add support for repetition to proc_macro::quote #141608

Conversation

moatom commented May 26, 2025 • edited by tgross35 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

moatom May 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tgross35 Jun 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ora-0 May 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ora-0 Jun 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ora-0 Jun 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rustbot commented Jun 1, 2025

Uh oh!

This comment has been minimized.

moatom commented Jun 1, 2025

Uh oh!

moatom commented Jun 1, 2025

Uh oh!

tgross35 commented Jun 1, 2025

Uh oh!

This comment has been minimized.

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tgross35 commented Jun 13, 2025

Uh oh!

moatom commented Jun 16, 2025

Uh oh!

dtolnay commented Jun 16, 2025

Uh oh!

bors commented Jun 16, 2025

Uh oh!

tgross35 commented Jun 16, 2025

Uh oh!

Uh oh!

Uh oh!

Add support for repetition to `proc_macro::quote` #141608

Add support for repetition to `proc_macro::quote` #141608

moatom commented May 26, 2025 •

edited by tgross35

Loading

moatom May 31, 2025 •

edited

Loading

tgross35 Jun 13, 2025 •

edited

Loading

ora-0 May 31, 2025 •

edited

Loading

ora-0 Jun 1, 2025 •

edited

Loading

ora-0 Jun 1, 2025 •

edited

Loading