-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Macros 1.2: Fast-track to stabilize function-like procedural macros #1913
Conversation
text/0000-macros-1.2.md
Outdated
|
||
The term *procedural macro* is somewhat ambiguous. | ||
Once upon a time, it was sometimes used for “old-style” compiler plugins. | ||
Such usage should be avoided, in favor of *compiler plugin* or *syntax extension*. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i thought we had already agreed that this was still a procedural macro. idk.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I didn’t know that. If the terminology is already agreed upon please let me know the details of it and I’ll update the RFC tomorrow.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is a rough consensus around procedural macro (c.f. declarative macro) for macros 2.0 and [compiler] plugin (preferred) or syntax extension for the legacy system.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That’s what I tried to say there. I’ll see if I can rephrase it to make it clearer.
And used (in a separate crate that depends on the previous one) like this: | ||
|
||
```rust | ||
foo!(...); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are there any constraints as to what these can expand to? Items/statements/expressions/patterns/etc?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
All of the above?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
More seriously: unless someone thinks otherwise, a starting point might be "same as macro_rules
".
1.1 documented some big use cases that justified a rush to stable, what specific big libraries would be helped by 1.2? |
This isn't even implemented yet... |
@Eh2406 Depending on how big is considered big, rust-phf is one. html5ever and cssparser also currently have big ugly build scripts that could be replaced by this, though its only for internal usage. I’ve had several ideas for crates (or features in existing crates) that I delayed because of this. Yes, the motivation is not as pressing as for custom derive. But as noted in the RFC, this is also a very small addition on top of what’s already stable. I think the trade-off is worth it, and this thread is all about discussing that. @abonander I hear from @jseyfried that it’s close. Also I don’t think implementation needs to be done before an RFC can be submitted to discuss something? |
It's my understanding that we want to hold off on stabilizing these until we have a better story WRT hygiene, but that's not official. Actually I was looking into implementing it but I was waiting for some PRs to be merged that dealt with some stuff first. |
hmmm.....my slippery slope alarm bells are going off, sorry. I'm worried about a "first mover effect" where good hygienic macros will forever be a smaller part of the ecosystem due to their later stabilization. Macros 1.1 was, IIUC, a special exception to our normal way of doing things due to the prominence of Serde and Diesel. I don't want to normalize it. |
text/0000-macros-1.2.md
Outdated
# Detailed design | ||
[design]: #detailed-design | ||
|
||
As a reminder, Macro 1.1 stabilized a new `proc_macros` crate with a very small public API: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: the crate's name is proc_macro
text/0000-macros-1.2.md
Outdated
|
||
The term *procedural macro* is somewhat ambiguous. | ||
Once upon a time, it was sometimes used for “old-style” compiler plugins. | ||
Such usage should be avoided, in favor of *compiler plugin* or *syntax extension*. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is a rough consensus around procedural macro (c.f. declarative macro) for macros 2.0 and [compiler] plugin (preferred) or syntax extension for the legacy system.
text/0000-macros-1.2.md
Outdated
|
||
* Terminology: *function-like procedural macro* is a mouthful. | ||
Is *function procedural macro* an acceptable approximation? | ||
*Functional procedural macro*? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would call these [normal|bang|function] procedural macros (c.f. attribute [procedural] macros, derive [procedural] macros).
text/0000-macros-1.2.md
Outdated
|
||
However this requires stabilizing `Delimiter` | ||
(including the presence or not of a `None` variant, for example), | ||
which is contrary to the goal of this RFC to stabilize as little as possible. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We could allow a Delimiter
parameter without requiring it, e.g.
// We could stabilize this:
#[proc_macro]
fn f(tokens: TokenStream) -> TokenStream { ... }
// and also support this later:
#[proc_macro]
fn f(delim: Delimiter, tokens: TokenStream) -> TokenStream { ... }
text/0000-macros-1.2.md
Outdated
* Change the `input: TokenStream` parameter to include the braces. | ||
In the first example above: `input.to_string() == "(...)"`. | ||
However this requires every macros that don’t care about the style of braces (most of them?) | ||
to have additional code to remove the braces in order to access the rest of the input. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If people want this, we could support it backwards compatibly via:
#[proc_macro]
fn f(input: TokenTree) -> TokenStream { ... }
I cannot easily see that Macros 1.2 would improve the situation here. The "big ugly script" will be still required for the procedural macro, so ergonomics do not really change IMO. To me valid and more urgent use cases for procedural function-like macros are those that require the token inspection, like |
This isn't a big use case and it isn't even yet released (just throwing out there that use cases exist!), but I'm currently working on a library that could benefit from this. In a nutshell my library is an alternative to let name = "Bob";
let numbers = &[1, 2, 3, 4];
let printer = |value: i32| { console.log("{}", value); };
js! {
console.log("Hello " + @{name});
var sum = @{numbers}.reduce(function(sum, value) { return sum + value; });
var cb = @{printer};
cb(sum);
cb.drop();
}; The snippet I've pasted works right now; I've implemented the |
One of the advantages I can see this bring is the generation from identifiers from function like macros. Right now this is not possible and all identifiers that a macro uses need to be passed directly to the macro (see rust-lang/rust#29599). However, I do agree with @Ericson2314 that this move will cause Macros 2.0 to always be used to a lesser extent for a long time. I'm not sure this is worth the the advantages I see at the moment. |
Regarding hygiene, @lifthrasiir I wrote these build scripts, I’m pretty sure they can be entirely replaced with proc macros and the result will be much nicer (less fragile). I’m referring here to the part where it parses an entire source file, walk over |
@SimonSapin Even item macros have to deal with hygienic identifiers being passed in, although the risk is smaller than supporting all possible macro positions, because there's no hygienic lexical scope to access, but, and here's the problematic part: you could pass the same identifier with different hygiene information (e.g. different "stack frames" of macro invocations) and the macro would lose the distinction. |
@eddyb Ok… so what does that say about the behavior that |
@SimonSapin I think that impl will always be a potential hygiene hole in procedural macros, the key distinction is that if we stabilize this now, more people will write procedural macros using that unhygienic system. |
I believe that @sgrif had some use cases for this in Diesel, no? I remember us discussing it when we were talking about Macros 1.1 and whether |
Does the Rustlang team have a ballpark guess on when a Macros 2.0 implementation will be started, let alone hit nightly? Macros 1.1 has already made Rust libraries like Serde and Diesel massively more usable, approaching high level scripting languages in ergonomics without sacrificing control or performance. Even a limited proc macro 1.2 would take that to a whole new level and make crates like Rocket and Stateful into first class citizens on stable. I believe this proposal's benefits outweigh the costs because it will make possible many more libraries with a level of expressiveness programmers expect from modern dynamic languages. Given that one of Rust's primary criticisms is its youth and ecosystem maturity, I think having this feature will rapidly increase language adoption among non-low-level programmers even if only a few library authors define proc macros directly. By the time Macros 2.0 hits, we will not only know a lot more about how proc macros are used in the wild but the compounding effects of adoption will benefit the entire community. Unlike features that solve specific pain points like non-lexical scoping or "impl Trait", proc macros are very visible to developers investigating a new language and seeing something like a stable async/await or easy Flask-like web framework implemented using macros could very well tip the scale for many of today's developers. I understand that there are issues with hygiene and concerns that this will undercut Macros 2.0 usage but I think there is probably a way to mitigate that without breaking backwards compatibility or locking Rust into a suboptimal but "good enough" design. Would it be possible to force these proc macros to be in a separate crate like derive macros and only allow them to be used from inside regular macro_rules! definitions? By adding an extra step to the Macros 1.0 expansion process, it might be possible to implement hygiene checks against the nested TokenStream output, provide slightly more helpful errors, and force library authors to expose Macros 1.2 in a way that can be easily replaced by Macros 2.0 without breaking public API backwards compatibility (as long as Macros 2.0 is as capable as these proc macros sans hygiene restrictions, which should be the case ). I'm sure there are some unintended consequences with this approach, if it is even possible, and it feels like a bit of a hack, but as I said above I believe the benefits are huge. (Edited because I'm unsure if this approach is possible without doing a deep dive into rustc) In my current side project (GUI framework backed by Webrender + Tokio), this functionality would be game changing. Proc macros would eliminate 80-90% of the boilerplate the framework user would have to deal with in stable and allow ergonomic async/generators (using reserved keywords like await and yield), expressive type checked state machines, runtime reflection/dynamic types, LINQ style DSLs for collections/iteratirs, and a zero-cost INotifyChanged (like in C#) interface. Some of those can be done with derive or macros by example but the implementations are fragile and far from ergonomic. |
So I'm certainly sympathetic. I feel like it'd be great to "unlock" I feel like I'd particularly like to support things like the ability to write class declarations that get mapped to JS or GNOME or whatever metadata. |
The use cases I'm interested in function-like procedural macros are also in the item position. Seems like a possibility that scoping this to the item level doesn't have such serious hygiene issues? |
I think it’s possible to hack something with derive in item position. I haven’t tried it though: macro_rules! foo {
($(tts:tt)*) => {
#[derive(Foo)]
#[foo($(tts)*)]
#[allow(dead_code)]
struct Dummy;
}
} #[proc_macro_derive(Foo, attributes(foo))]
pub fn foo(input: TokenStream) -> TokenStream {
// Extract the foo attribute with syn
// Return new items
} |
I think hygiene is futile with a String → String transformation. If we need hygiene the TokenStream API need more than Yet how many procedural function-like macros (instead of declarative macros) requires hygiene? I interpret this as "blindly pasting the output string into the token stream is going to cause problem". Just by checking the reverse dependencies of syntex_syntax and crates named "codegen" which provides function-like macros, I think none of them requires hygiene.
Besides function-like macros, function/struct attributes seems to be used for some big crates. They are also unhygienic.
That said, I don't think we need to rush into stablizing function procedural macros, as workaround exists. |
Thanks for the RFC @SimonSapin! This was actually historically included in macros 1.1, but @nrc convinced me personally at least that it was worthwhile to leave out. I'm personally in favor and would be totally ok seeing it land! One thing I'd like to see is a few more details in the detailed design section as well. For example what precisely is the I personally would love to see an accompanying implementation as well as that typically raises lots of questions that fill out the detailed design section nicely as well, but I understand if that's too much to ask. In this case though it may not be too hard as macros 1.1 laid all the groundwork! |
Should the macro get information on where it is being used ? eg is it in item/expr/pattern/... position |
Yes. This would be huge for us. |
I'm a little dismayed that progress is perceived to be so slow that another hacky, broken "fast track" API is desired on top of the first one. We should just get the ball rolling on procedural macros 2.0 proper. There aren't nearly as many open design questions as with declarative macros, and I don't think it's really all that difficult to get hygiene and line info working right. |
|
||
Terminology: | ||
|
||
* *Function procedural macro*: a function declared with the `proc_macro` attribute. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Probably an insignificant gripe, but I find that the following sounds more natural to me (as a native English speaker):
- Procedural function macro.
- Procedural attribute macro.
- Procedural derive macro.
Essentially "procedural <something> macro" versus "<something> procedural macro".
Does anyone else think the same?
Can you expand on what you would want?
Actually, my impression is that progress here has been quite rapid. It'd be helpful for sure to have @jseyfried summarize the "current state of play". That said, I think that some details of a newer system -- notably an improved hygiene system that accounts not only for local variables but privacy, method scopes, and other sorts of things -- have a lot of complications and will take some time to stabilize. If we can find other niches to "carve out" that don't depend on that, it seems like a win to me. (For that matter, I think that if people are writing a bunch of "string-based" procedural macros and we then include a Better Way To Do It, that supports hygiene and offers new capabilities (e.g., access to private data), I would expect that people will upgrade to the newer system in any case.) |
@nikomatsakis The scary part is breaking users when upgrading the implementation to consider hygiene. |
@nikomatsakis We have 3 non-derive proc macros in Diesel. macro_rules! infer_schema {
($database_url:expr) => {
mod __diesel_infer_schema {
#[derive(InferSchema)]
#[infer_schema_arguments(database_url=$database_url)]
struct _Dummy;
}
pub use __diesel_infer_schema::*;
}
} The limitations of this approach (compared to when this was implemented with
Any implementation of Macros 1.2 would remove the first limitation. But for the system to let us have the capabilities that the legacy |
It's my opinion, as mostly a Rust lurker, that Macros 2.0 should be the focus. |
Arguably, hygienic Well, the Common Lisp community would be the big counter example about people upgrading to better macros once they become available. Sadly, Hygiene evidently isn't always appreciated by those used to working without it. We need to be very careful with teaching and usability then.
Privacy applies to items, and even derive impls. Methods will work like associated types with the new system, and associated types clearly are relevant to items. Last I talked to @jseyfried all this is indeed was going well, and the design was mostly figured out. (At least I was very OK with it, and I didn't see many axes worth taking another route.) Let's at least wait for that work to hit nightly and then talk about stabilizing a subset of what's already been implemented. |
There could be a But fn plus(a: TokenStream) -> TokenStream {
TokenStream::from_hygienic_str(&format!("{{ let x = 1; x + {} }}", a.to_string())).unwrap()
} What to do if your input is Enforcing hygiene with a String is a wrong approach, it should be done on the token level where each identifier token can carry the SyntaxContext info. (As I mentioned in my comment above, I don't think procedural macros today require or want hygiene in its output.) |
I strongly object to this. Let me first clarify why I think macros 1.1 (custom derive was a good idea to propose):
None of these are the case for this proposal.
|
And a more positive point: we expect significant progress on procedural macros this year. There are significant open design questions (notably around hygiene), but @jseyfried has been doing excellent work laying down the foundations for this feature and I expect to see a functioning implementation of the new system well before the end of the year. (Although stability will take longer, as usual). |
Huh, so I have to say that this hack that @sgrif describes is pretty clever, and could probably be used to fulfill most of the use-cases I have in mind. This inclines me more to @nrc's point-of-view -- given the rapid progress that @jseyfried has been making, we probably can afford to wait and get more time to "do it right". |
True, but the scope/hygiene for local variables from
Sadly, this isn't exactly true. #![feature(use_extern_macros)] // or `#![feature(proc_macro)]`, which implies this
// crate A:
pub struct Z;
#[macro_export]
macro_rules! m { () => {
#[derive(Custom)]
struct Y($crate::Z);
} }
// crate B:
extern crate A;
pub use A::m;
// crate C:
extern crate B;
B::m!();
//^ This expands to `#[derive(Custom)] struct Y($crate::Z);`.
//| We can't stringify `$crate::Z` to resolve correctly to `Z` from crate `A`. Today, we make an effort (not yet best effort) to stringify correctly.
Not much -- I think there are a couple of reasonable points in the design space. For example, we could emit an ambiguity error if there is more than once correct choice for hygiene of a
Macros 2.0 implementation is under way (see the roadmaps for macro modularization, procedural macros, and declarative macros). More
That doesn't work in general since attributes do not allow arbitrary tokens. However, this should work on nightly once I support arbitrary tokens in attributes next week, and we could fast-track that to stable.
Not as an argument to the
Indeed, although I've been prioritizing unrelated groundwork for procedural macros 2.0. That being said, the hygiene stuff should hit nightly via declarative macros 2.0 sometime in March. |
That would be nice. I’ve landed an approximation of the extern crate phf;
#[macro_use] extern crate cssparser;
#[macro_use] extern crate cssparser_macros;
fn color_rgb(input: &str) -> Option<(u8, u8, u8)> {
ascii_case_insensitive_phf_map! {
KEYWORDS: Map<(u8, u8, u8)> = {
"red" => "(255, 0, 0)",
"green" => "(0, 255, 0)",
"blue" => "(0, 0, 255)",
}
}
KEYWORDS::get(input).cloned()
} |
@SimonSapin the proc-macro-hack way gives you arbitrary tokens. 😈 #[derive(cssparser__phf_map)]
enum Dummy {
Input = (stringify!($($tt)*), 0).1
} |
@kennytm the short answer is quasi-quoting avoids that, and needing to convert to strings altogether. I suspect there will be no use for going to and from strings, so might as well make them correct. The medium answer is if we could concat and "wrap" token streams, then we can do things in a way almost as good as quasi-quoting, and with hygenic from_ fn plus(a: TokenStream) -> TokenStream {
TokenStream::from_str("let x = 1; x +").unwrap()
.concat(a)
.wrap('{') // make new tree node: { ...everything so far... }
} I agree explicit renaming is awful. I say the solution is just never go to strings, but only from them (and only from them hygienically). [If you are wondering what to do about |
I’ve very happy with the trick shown by @dtolnay above. As an alternative to his One point that improves the ergonomics of all this for library users is that Also, thanks to the discussions here I understand hygiene better and am somewhat convinced that stabilizing For these reasons combined, I am hereby rescinding this RFC. (Though if someone else wants to champion it feel free, I can re-open.) |
It would be even better without rust-lang/rust#39889! |
rust-lang/rust#39889 will be fixed once rust-lang/rust#39419 lands. |
Implement function-like procedural macros ( `#[proc_macro]`) Adds the `#[proc_macro]` attribute, which expects bare functions of the kind `fn(TokenStream) -> TokenStream`, which can be invoked like `my_macro!()`. cc rust-lang/rfcs#1913, rust-lang#38356 r? @jseyfried cc @nrc
Implement function-like procedural macros ( `#[proc_macro]`) Adds the `#[proc_macro]` attribute, which expects bare functions of the kind `fn(TokenStream) -> TokenStream`, which can be invoked like `my_macro!()`. cc rust-lang/rfcs#1913, rust-lang#38356 r? @jseyfried cc @nrc
Implement function-like procedural macros ( `#[proc_macro]`) Adds the `#[proc_macro]` attribute, which expects bare functions of the kind `fn(TokenStream) -> TokenStream`, which can be invoked like `my_macro!()`. cc rust-lang/rfcs#1913, rust-lang#38356 r? @jseyfried cc @nrc
Stabilize function-like procedural macros (whose usage looks like
foo!(...)
), like this was done in “Macros 1.1” for customderive
, before “Macros 2.0” is fully ready.Rendered