Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Macros 1.2: Fast-track to stabilize function-like procedural macros #1913

Closed
wants to merge 4 commits into from

Conversation

SimonSapin
Copy link
Contributor

Stabilize function-like procedural macros (whose usage looks like foo!(...)), like this was done in “Macros 1.1” for custom derive, before “Macros 2.0” is fully ready.

Rendered


The term *procedural macro* is somewhat ambiguous.
Once upon a time, it was sometimes used for “old-style” compiler plugins.
Such usage should be avoided, in favor of *compiler plugin* or *syntax extension*.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i thought we had already agreed that this was still a procedural macro. idk.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn’t know that. If the terminology is already agreed upon please let me know the details of it and I’ll update the RFC tomorrow.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a rough consensus around procedural macro (c.f. declarative macro) for macros 2.0 and [compiler] plugin (preferred) or syntax extension for the legacy system.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That’s what I tried to say there. I’ll see if I can rephrase it to make it clearer.

And used (in a separate crate that depends on the previous one) like this:

```rust
foo!(...);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there any constraints as to what these can expand to? Items/statements/expressions/patterns/etc?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All of the above?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More seriously: unless someone thinks otherwise, a starting point might be "same as macro_rules".

@Eh2406
Copy link
Contributor

Eh2406 commented Feb 20, 2017

1.1 documented some big use cases that justified a rush to stable, what specific big libraries would be helped by 1.2?

@abonander
Copy link

This isn't even implemented yet...

@SimonSapin
Copy link
Contributor Author

@Eh2406 Depending on how big is considered big, rust-phf is one. html5ever and cssparser also currently have big ugly build scripts that could be replaced by this, though its only for internal usage. I’ve had several ideas for crates (or features in existing crates) that I delayed because of this.

Yes, the motivation is not as pressing as for custom derive. But as noted in the RFC, this is also a very small addition on top of what’s already stable. I think the trade-off is worth it, and this thread is all about discussing that.

@abonander I hear from @jseyfried that it’s close. Also I don’t think implementation needs to be done before an RFC can be submitted to discuss something?

@abonander
Copy link

abonander commented Feb 20, 2017

It's my understanding that we want to hold off on stabilizing these until we have a better story WRT hygiene, but that's not official.

Actually I was looking into implementing it but I was waiting for some PRs to be merged that dealt with some stuff first.

@Ericson2314
Copy link
Contributor

hmmm.....my slippery slope alarm bells are going off, sorry. I'm worried about a "first mover effect" where good hygienic macros will forever be a smaller part of the ecosystem due to their later stabilization. Macros 1.1 was, IIUC, a special exception to our normal way of doing things due to the prominence of Serde and Diesel. I don't want to normalize it.

@aturon aturon added the T-lang Relevant to the language team, which will review and decide on the RFC. label Feb 20, 2017
@aturon
Copy link
Member

aturon commented Feb 20, 2017

# Detailed design
[design]: #detailed-design

As a reminder, Macro 1.1 stabilized a new `proc_macros` crate with a very small public API:

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: the crate's name is proc_macro


The term *procedural macro* is somewhat ambiguous.
Once upon a time, it was sometimes used for “old-style” compiler plugins.
Such usage should be avoided, in favor of *compiler plugin* or *syntax extension*.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a rough consensus around procedural macro (c.f. declarative macro) for macros 2.0 and [compiler] plugin (preferred) or syntax extension for the legacy system.


* Terminology: *function-like procedural macro* is a mouthful.
Is *function procedural macro* an acceptable approximation?
*Functional procedural macro*?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would call these [normal|bang|function] procedural macros (c.f. attribute [procedural] macros, derive [procedural] macros).


However this requires stabilizing `Delimiter`
(including the presence or not of a `None` variant, for example),
which is contrary to the goal of this RFC to stabilize as little as possible.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could allow a Delimiter parameter without requiring it, e.g.

// We could stabilize this:
#[proc_macro]
fn f(tokens: TokenStream) -> TokenStream { ... }

// and also support this later:
#[proc_macro]
fn f(delim: Delimiter, tokens: TokenStream) -> TokenStream { ... }

* Change the `input: TokenStream` parameter to include the braces.
In the first example above: `input.to_string() == "(...)"`.
However this requires every macros that don’t care about the style of braces (most of them?)
to have additional code to remove the braces in order to access the rest of the input.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If people want this, we could support it backwards compatibly via:

#[proc_macro]
fn f(input: TokenTree) -> TokenStream { ... }

@lifthrasiir
Copy link
Contributor

@SimonSapin

html5ever and cssparser also currently have big ugly build scripts that could be replaced by this, though its only for internal usage. I’ve had several ideas for crates (or features in existing crates) that I delayed because of this.

I cannot easily see that Macros 1.2 would improve the situation here. The "big ugly script" will be still required for the procedural macro, so ergonomics do not really change IMO. To me valid and more urgent use cases for procedural function-like macros are those that require the token inspection, like format!.

@koute
Copy link
Member

koute commented Feb 21, 2017

1.1 documented some big use cases that justified a rush to stable, what specific big libraries would be helped by 1.2?

This isn't a big use case and it isn't even yet released (just throwing out there that use cases exist!), but I'm currently working on a library that could benefit from this. In a nutshell my library is an alternative to webplatform where you can mostly seamlessly embed JavaScript in your code like this:

let name = "Bob";
let numbers = &[1, 2, 3, 4];
let printer = |value: i32| { console.log("{}", value); };

js! {
    console.log("Hello " + @{name});
    var sum = @{numbers}.reduce(function(sum, value) { return sum + value; });
    var cb = @{printer};
    cb(sum);
    cb.drop();
};

The snippet I've pasted works right now; I've implemented the js! macro with plain old macro_rules!, however as you might expect it is pretty much an eldritch monstrosity. Having procedural macros would allow me to make the implementation a lot nicer, make it possible to verify the syntax of the passed JavaScript (as opposed to blowing up at runtime due to a missing semicolon) and wouldn't require the users to increase the compiler recursion limit through the roof just to be able to use this macro.

@JelteF
Copy link

JelteF commented Feb 21, 2017

One of the advantages I can see this bring is the generation from identifiers from function like macros. Right now this is not possible and all identifiers that a macro uses need to be passed directly to the macro (see rust-lang/rust#29599).

However, I do agree with @Ericson2314 that this move will cause Macros 2.0 to always be used to a lesser extent for a long time. I'm not sure this is worth the the advantages I see at the moment.

@SimonSapin
Copy link
Contributor Author

Regarding hygiene, FromStr for TokenStream is already stable, so it will still need to be supported when Macros 2.0 are fully implemented. And it will need to deal with hygiene in a way that doesn’t break current versions of Serde and other derive proc macros. I don’t really understand how this RFC makes hygiene concerns any worse than they already are.

@lifthrasiir I wrote these build scripts, I’m pretty sure they can be entirely replaced with proc macros and the result will be much nicer (less fragile). I’m referring here to the part where it parses an entire source file, walk over TokenTree to find Token("foo"), Token("!"), Tree("[]", […]), create another TokenTree with that expanded, and serialize to a file. Then the crate uses that generated file with include!.

@eddyb
Copy link
Member

eddyb commented Feb 21, 2017

@SimonSapin #[derive] has the amazing property of not having any hygienic inputs. Well, you could put local variables inside array lengths, but they wouldn't compile. That's all true in the current system, anyway.

Even item macros have to deal with hygienic identifiers being passed in, although the risk is smaller than supporting all possible macro positions, because there's no hygienic lexical scope to access, but, and here's the problematic part: you could pass the same identifier with different hygiene information (e.g. different "stack frames" of macro invocations) and the macro would lose the distinction.

@SimonSapin
Copy link
Contributor Author

@eddyb Ok… so what does that say about the behavior that FromStr for TokenStream should have in a proc_macro when everything will be implemented?

@withoutboats
Copy link
Contributor

@SimonSapin I think that impl will always be a potential hygiene hole in procedural macros, the key distinction is that if we stabilize this now, more people will write procedural macros using that unhygienic system.

@nikomatsakis
Copy link
Contributor

I believe that @sgrif had some use cases for this in Diesel, no? I remember us discussing it when we were talking about Macros 1.1 and whether #[derive] impls could "rewrite" the struct entirely.

@akiselev
Copy link

akiselev commented Feb 21, 2017

Does the Rustlang team have a ballpark guess on when a Macros 2.0 implementation will be started, let alone hit nightly? Macros 1.1 has already made Rust libraries like Serde and Diesel massively more usable, approaching high level scripting languages in ergonomics without sacrificing control or performance. Even a limited proc macro 1.2 would take that to a whole new level and make crates like Rocket and Stateful into first class citizens on stable. I believe this proposal's benefits outweigh the costs because it will make possible many more libraries with a level of expressiveness programmers expect from modern dynamic languages. Given that one of Rust's primary criticisms is its youth and ecosystem maturity, I think having this feature will rapidly increase language adoption among non-low-level programmers even if only a few library authors define proc macros directly. By the time Macros 2.0 hits, we will not only know a lot more about how proc macros are used in the wild but the compounding effects of adoption will benefit the entire community. Unlike features that solve specific pain points like non-lexical scoping or "impl Trait", proc macros are very visible to developers investigating a new language and seeing something like a stable async/await or easy Flask-like web framework implemented using macros could very well tip the scale for many of today's developers.

I understand that there are issues with hygiene and concerns that this will undercut Macros 2.0 usage but I think there is probably a way to mitigate that without breaking backwards compatibility or locking Rust into a suboptimal but "good enough" design. Would it be possible to force these proc macros to be in a separate crate like derive macros and only allow them to be used from inside regular macro_rules! definitions? By adding an extra step to the Macros 1.0 expansion process, it might be possible to implement hygiene checks against the nested TokenStream output, provide slightly more helpful errors, and force library authors to expose Macros 1.2 in a way that can be easily replaced by Macros 2.0 without breaking public API backwards compatibility (as long as Macros 2.0 is as capable as these proc macros sans hygiene restrictions, which should be the case ). I'm sure there are some unintended consequences with this approach, if it is even possible, and it feels like a bit of a hack, but as I said above I believe the benefits are huge. (Edited because I'm unsure if this approach is possible without doing a deep dive into rustc)

In my current side project (GUI framework backed by Webrender + Tokio), this functionality would be game changing. Proc macros would eliminate 80-90% of the boilerplate the framework user would have to deal with in stable and allow ergonomic async/generators (using reserved keywords like await and yield), expressive type checked state machines, runtime reflection/dynamic types, LINQ style DSLs for collections/iteratirs, and a zero-cost INotifyChanged (like in C#) interface. Some of those can be done with derive or macros by example but the implementations are fragile and far from ergonomic.

@nikomatsakis
Copy link
Contributor

So I'm certainly sympathetic. I feel like it'd be great to "unlock" foo! macros. But the last time we talked about it, I know that @nrc raised a number of objections relating to hygiene. I am though somewhat leery of blocking on a new hygiene system, which feels like a complex undertaking. I don't know if there's a middle-ground -- i.e., maybe supporting foo! macros but only in item position or something? (As opposed to for expressions.)

I feel like I'd particularly like to support things like the ability to write class declarations that get mapped to JS or GNOME or whatever metadata.

@withoutboats
Copy link
Contributor

The use cases I'm interested in function-like procedural macros are also in the item position. Seems like a possibility that scoping this to the item level doesn't have such serious hygiene issues?

@SimonSapin
Copy link
Contributor Author

I think it’s possible to hack something with derive in item position. I haven’t tried it though:

macro_rules! foo {
    ($(tts:tt)*) => {
        #[derive(Foo)]
        #[foo($(tts)*)]
        #[allow(dead_code)]
        struct Dummy;
    }
}
#[proc_macro_derive(Foo, attributes(foo))]
pub fn foo(input: TokenStream) -> TokenStream {
    // Extract the foo attribute with syn

    // Return new items
}

@kennytm
Copy link
Member

kennytm commented Feb 21, 2017

I think hygiene is futile with a String → String transformation. If we need hygiene the TokenStream API need more than to_string() and from_str().

Yet how many procedural function-like macros (instead of declarative macros) requires hygiene? I interpret this as "blindly pasting the output string into the token stream is going to cause problem". Just by checking the reverse dependencies of syntex_syntax and crates named "codegen" which provides function-like macros, I think none of them requires hygiene.

  • cpp (cpp!) — Outputs several items (and a static library), no hygiene.
  • dotenv (dotenv!) — Takes one or two string literals, hygiene irrelevant
  • extprim_literals (u128!, i128!) — Take an integer literal, output struct literal, hygiene irrelevant
  • indoc (indoc!) — Take (byte-)string literal, output (byte-)string literal, hygiene irrelevant
  • nametable_codegen (nametable!) — Output modules of with name given in input, no hygiene.
  • rustlex (rustlex!) — Output a lexer struct of the given name, no hygiene.
  • quasi (quasi_xxxx!) — Not sure, but crate is irrelevant in String → String transformation.
  • synthax (quote_xxxx!) — Not sure, but crate is irrelevant in String → String transformation.
  • easy-plugin (easy_plugin!) — Output a function of given name, no hygiene.
  • phf_macros (phf_map!) — Takes DSL input, output struct literal, hygiene irrelevant

Besides function-like macros, function/struct attributes seems to be used for some big crates. They are also unhygienic.

  • pnet_macros#[packet] on struct, should probably become #[derive(Packet)] though?
  • borealis#[template_xxxx(...)] on struct.
  • rocket#[route], #[get], #[post] etc on function.
  • init#[init] on function.

That said, I don't think we need to rush into stablizing function procedural macros, as workaround exists.

@alexcrichton
Copy link
Member

Thanks for the RFC @SimonSapin! This was actually historically included in macros 1.1, but @nrc convinced me personally at least that it was worthwhile to leave out. I'm personally in favor and would be totally ok seeing it land!

One thing I'd like to see is a few more details in the detailed design section as well. For example what precisely is the TokenStream coming in? Does it include delimeters? What precisely is the token stream going out? Is it an item? an expression? (there are multiple possibilities here today)

I personally would love to see an accompanying implementation as well as that typically raises lots of questions that fill out the detailed design section nicely as well, but I understand if that's too much to ask. In this case though it may not be too hard as macros 1.1 laid all the groundwork!

@plietar
Copy link

plietar commented Feb 21, 2017

Should the macro get information on where it is being used ? eg is it in item/expr/pattern/... position

@sgrif
Copy link
Contributor

sgrif commented Feb 22, 2017

I believe that @sgrif had some use cases for this in Diesel, no?

Yes. This would be huge for us.

@comex
Copy link

comex commented Feb 22, 2017

I'm a little dismayed that progress is perceived to be so slow that another hacky, broken "fast track" API is desired on top of the first one.

We should just get the ball rolling on procedural macros 2.0 proper. There aren't nearly as many open design questions as with declarative macros, and I don't think it's really all that difficult to get hygiene and line info working right.


Terminology:

* *Function procedural macro*: a function declared with the `proc_macro` attribute.
Copy link

@severen severen Feb 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably an insignificant gripe, but I find that the following sounds more natural to me (as a native English speaker):

  • Procedural function macro.
  • Procedural attribute macro.
  • Procedural derive macro.

Essentially "procedural <something> macro" versus "<something> procedural macro".

Does anyone else think the same?

@nikomatsakis
Copy link
Contributor

@sgrif

Yes. This would be huge for us.

Can you expand on what you would want?


@comex

I'm a little dismayed that progress is perceived to be so slow that another hacky, broken "fast track" API is desired on top of the first one.

Actually, my impression is that progress here has been quite rapid. It'd be helpful for sure to have @jseyfried summarize the "current state of play". That said, I think that some details of a newer system -- notably an improved hygiene system that accounts not only for local variables but privacy, method scopes, and other sorts of things -- have a lot of complications and will take some time to stabilize. If we can find other niches to "carve out" that don't depend on that, it seems like a win to me.

(For that matter, I think that if people are writing a bunch of "string-based" procedural macros and we then include a Better Way To Do It, that supports hygiene and offers new capabilities (e.g., access to private data), I would expect that people will upgrade to the newer system in any case.)

@eddyb
Copy link
Member

eddyb commented Feb 22, 2017

@nikomatsakis The scary part is breaking users when upgrading the implementation to consider hygiene.

@sgrif
Copy link
Contributor

sgrif commented Feb 22, 2017

@nikomatsakis We have 3 non-derive proc macros in Diesel. infer_schema!, infer_table_from_schema!, and embed_migrations!. We basically fake it right now by doing:

macro_rules! infer_schema {
    ($database_url:expr) => {
        mod __diesel_infer_schema {
            #[derive(InferSchema)]
            #[infer_schema_arguments(database_url=$database_url)]
            struct _Dummy;
        }
        pub use __diesel_infer_schema::*;
    }
}

The limitations of this approach (compared to when this was implemented with plugin_registrar) are:

  • The macro cannot be invoked more than once unless the invocation is wrapped in another module.
  • The macro cannot receive any arguments other than a literal. This means that infer_schema!(env!("DATABASE_URL")) no longer works. We provide some "magic" for the common cases by allowing things like infer_schema!("env:DATABASE_URL"), but a common request is that people have their own config files that they want to use. They can't just compose in their own macro as an argument. It's also much harder to document and discover.

Any implementation of Macros 1.2 would remove the first limitation. But for the system to let us have the capabilities that the legacy plugin_registrar system gave us, we would need any macros in the input token stream to be expanded before they are given to us, or some way to explicitly request that macro expansion be done on a token stream.

@kylone
Copy link

kylone commented Feb 22, 2017

It's my opinion, as mostly a Rust lurker, that Macros 2.0 should be the focus. string -> string macros have pretty significant drawback. I'm just recommend being very public about Macros 2.0 work. There's a significant amount of community urgency procedural macros.

@Ericson2314
Copy link
Contributor

Ericson2314 commented Feb 22, 2017

Arguably, hygienic FromString for TokenStream would introduce tokens that, when parsed, would only include fresh / non-capturing identifiers. (e.g. { let x = 1; x} is fine because the binding is part of the string, but Foo::asdf always will give have Foo and asdf unbound). This, I assume, would break all 1.1 macros.


@nikomatsakis

Well, the Common Lisp community would be the big counter example about people upgrading to better macros once they become available. Sadly, Hygiene evidently isn't always appreciated by those used to working without it. We need to be very careful with teaching and usability then.

privacy, method scopes

Privacy applies to items, and even derive impls. Methods will work like associated types with the new system, and associated types clearly are relevant to items.


Last I talked to @jseyfried all this is indeed was going well, and the design was mostly figured out. (At least I was very OK with it, and I didn't see many axes worth taking another route.) Let's at least wait for that work to hit nightly and then talk about stabilizing a subset of what's already been implemented.

@kennytm
Copy link
Member

kennytm commented Feb 23, 2017

@Ericson2314

There could be a TokenStream::from_hygienic_str, the meaning of the existing FromStr does not need to be changed.

But TokenStream::from_hygienic_str alone is not going to help if your macro is like

fn plus(a: TokenStream) -> TokenStream {
    TokenStream::from_hygienic_str(&format!("{{ let x = 1; x + {} }}", a.to_string())).unwrap()
}

What to do if your input is x? You would need to provide a .to_hygienic_string() as well to produce something like x$180a8d2c, and then TokenStream::from_hygienic_str will need to rename those back to the normal names, which is a total mess.

Enforcing hygiene with a String is a wrong approach, it should be done on the token level where each identifier token can carry the SyntaxContext info.

(As I mentioned in my comment above, I don't think procedural macros today require or want hygiene in its output.)

@nrc
Copy link
Member

nrc commented Feb 23, 2017

I strongly object to this.

Let me first clarify why I think macros 1.1 (custom derive was a good idea to propose):

  • The motivation was super strong. Specifically, serialisation is a key feature of programming languages and the strongly preferred approach (pretty much the only serious approach) to serialisation in Rust involved custom derive. The language was basically crippled for a large number of users without this feature and there were no feasible alternatives.
  • For the very specific case of custom derive, hygiene does not really matter. There is basically no way to screw up by being unhygienic.
  • The design space is small and well-understood - there were not many alternatives to the custom derive design and we've been using derive and custom derive for years in unstable Rust and most people are pretty satisfied by the feature as it is being used.
  • The proposed feature (macros 1.1) is a clear subset of any long term solution.

None of these are the case for this proposal.

  • It would be nice to move macro users to stable Rust, but I don't see that the motivation here is any stronger than for any other proposed feature. This is equivalent to saying - lets just stabilise an unsound and half-designed version of specialisation or impl Trait, just because.
  • Hygiene is super important for macros. It is the fundamental property that makes them safe. Shrugging at hygiene is like shrugging at type safety. It is the difference between C macros where every usage makes a reviewer shudder and triple-check the code and Scheme macros where they're a core part of the language and part of the general work-flow. How do we want Rust macros to feel?
  • There is a huge design space for procedural macros. I have sketched an approach in an RFC, but there are a lot of details missing and we need experience with an implementation before we can be sure of the design. There are also many alternatives. The only thing I am sure about is that the existing implementation is bad and needs improvement.
  • Given that we don't know what the long-term solution will be, it seems impossible to say whether this will be a subset of it. However, I am pretty sure that string-manipluation will not be part of the long term plan, beyond perhaps a recommended-against appendix that will either be deprecated or somehow restricted to custom derive. Thus this is not only a step too fast, but a step in the wrong direction.

@nrc
Copy link
Member

nrc commented Feb 24, 2017

And a more positive point: we expect significant progress on procedural macros this year. There are significant open design questions (notably around hygiene), but @jseyfried has been doing excellent work laying down the foundations for this feature and I expect to see a functioning implementation of the new system well before the end of the year. (Although stability will take longer, as usual).

@nikomatsakis
Copy link
Contributor

Huh, so I have to say that this hack that @sgrif describes is pretty clever, and could probably be used to fulfill most of the use-cases I have in mind. This inclines me more to @nrc's point-of-view -- given the rapid progress that @jseyfried has been making, we probably can afford to wait and get more time to "do it right".

@jseyfried
Copy link

jseyfried commented Feb 27, 2017

@SimonSapin

Regarding hygiene, FromStr for TokenStream is already stable, so it will still need to be supported when Macros 2.0 are fully implemented

True, but the scope/hygiene for local variables from impl FromStr for TokenStream is not fully stable. impl FromStr for TokenStream can create local variables in an item on stable, but the local variables in scope at the invocation site are always blocked by the item, so they are not yet relevant.

@eddyb

#[derive] has the amazing property of not having any hygienic inputs

Sadly, this isn't exactly true. $crate is hygienic, and this can cause problems today. For example,

#![feature(use_extern_macros)] // or `#![feature(proc_macro)]`, which implies this

// crate A:
pub struct Z;

#[macro_export]
macro_rules! m { () => {
    #[derive(Custom)]
    struct Y($crate::Z);
} }

// crate B:
extern crate A;
pub use A::m;

// crate C:
extern crate B;
B::m!();
//^ This expands to `#[derive(Custom)] struct Y($crate::Z);`.
//| We can't stringify `$crate::Z` to resolve correctly to `Z` from crate `A`.

Today, we make an effort (not yet best effort) to stringify correctly.

@SimonSapin

so what does that say about the behavior that FromStr for TokenStream should have in a proc_macro when everything will be implemented

Not much -- I think there are a couple of reasonable points in the design space. For example, we could emit an ambiguity error if there is more than once correct choice for hygiene of a FromStr for TokenStream-generated variable.

@akiselev

Does the Rustlang team have a ballpark guess on when a Macros 2.0 implementation will be started, let alone hit nightly?

Macros 2.0 implementation is under way (see the roadmaps for macro modularization, procedural macros, and declarative macros). More TokenStream API should start landing by mid-March.

@SimonSapin

I think it’s possible to hack something with derive in item position (#1913 (comment))

That doesn't work in general since attributes do not allow arbitrary tokens. However, this should work on nightly once I support arbitrary tokens in attributes next week, and we could fast-track that to stable.

@plietar

Should the macro get information on where it is being used?

Not as an argument to the #[proc_macro] function (imo), but perhaps via the "macro context" (thread-local state). Regardless, we can add new #[proc_macro] function signatures backwards-compatibly later.

@Ericson2314

Last I talked to @jseyfried all this [hygiene design] was going well

Indeed, although I've been prioritizing unrelated groundwork for procedural macros 2.0. That being said, the hygiene stuff should hit nightly via declarative macros 2.0 sometime in March.

@SimonSapin
Copy link
Contributor Author

I think it’s possible to hack something with derive in item position

That doesn't work in general since attributes do not allow arbitrary tokens. However, this should work on nightly once I support arbitrary tokens in attributes next week, and we could fast-track that to stable.

That would be nice. I’ve landed an approximation of the phf_map!(…) macro (with ASCII-case-insensitivity, but that part can easily be factored out) that runs on stable, with a macro_rules! macro combined with a proc_macro_derive on a dummy type and a custom attribute on that type’s definition. I was hoping to use stringify!() to turn abitrary tokens (const expression that initialize map values in a static) into a string to be passed to an attribute, but it looks like the current check that attribute only have string parameters is done before stringify! is expanded, so that didn’t work. I ended up with string literals that contain Rust syntax:

extern crate phf;
#[macro_use] extern crate cssparser;
#[macro_use] extern crate cssparser_macros;

fn color_rgb(input: &str) -> Option<(u8, u8, u8)> {
    ascii_case_insensitive_phf_map! {
        KEYWORDS: Map<(u8, u8, u8)> = {
            "red" => "(255, 0, 0)",
            "green" => "(0, 255, 0)",
            "blue" => "(0, 0, 255)",
        }
    }
    KEYWORDS::get(input).cloned()
}

@dtolnay
Copy link
Member

dtolnay commented Feb 27, 2017

@SimonSapin the proc-macro-hack way gives you arbitrary tokens. 😈

#[derive(cssparser__phf_map)]
enum Dummy {
    Input = (stringify!($($tt)*), 0).1
}

@Ericson2314
Copy link
Contributor

Ericson2314 commented Feb 27, 2017

@kennytm the short answer is quasi-quoting avoids that, and needing to convert to strings altogether. I suspect there will be no use for going to and from strings, so might as well make them correct.

The medium answer is if we could concat and "wrap" token streams, then we can do things in a way almost as good as quasi-quoting, and with hygenic from_str only:

fn plus(a: TokenStream) -> TokenStream {
    TokenStream::from_str("let x = 1; x +").unwrap()
        .concat(a)
        .wrap('{') // make new tree node: { ...everything so far... } 
}

I agree explicit renaming is awful. I say the solution is just never go to strings, but only from them (and only from them hygienically).

[If you are wondering what to do about x + {} + x, were the xs on either side must be the same, I say still don't use to_string but instead arithmetically weaken hygiene by unifying variables. This would be done before the concatenation.]

@SimonSapin
Copy link
Contributor Author

I’ve very happy with the trick shown by @dtolnay above. As an alternative to his proc-macro-hack crate I made procedural-masquerade to help other crate maintainers write functional macros with a procedural component. And this works on Rust 1.15 stable!

One point that improves the ergonomics of all this for library users is that pub use some_proc_macro_crate::*; works on 1.15 stable with custom derives.


Also, thanks to the discussions here I understand hygiene better and am somewhat convinced that stabilizing #[proc_macro] without a story for hygiene is probably not a good idea.

For these reasons combined, I am hereby rescinding this RFC. (Though if someone else wants to champion it feel free, I can re-open.)

@SimonSapin SimonSapin closed this Feb 28, 2017
@dtolnay
Copy link
Member

dtolnay commented Feb 28, 2017

It would be even better without rust-lang/rust#39889!

@jseyfried
Copy link

rust-lang/rust#39889 will be fixed once rust-lang/rust#39419 lands.

frewsxcv added a commit to frewsxcv/rust that referenced this pull request Mar 2, 2017
Implement function-like procedural macros ( `#[proc_macro]`)

Adds the `#[proc_macro]` attribute, which expects bare functions of the kind `fn(TokenStream) -> TokenStream`, which can be invoked like `my_macro!()`.

cc rust-lang/rfcs#1913, rust-lang#38356

r? @jseyfried
cc @nrc
frewsxcv added a commit to frewsxcv/rust that referenced this pull request Mar 2, 2017
Implement function-like procedural macros ( `#[proc_macro]`)

Adds the `#[proc_macro]` attribute, which expects bare functions of the kind `fn(TokenStream) -> TokenStream`, which can be invoked like `my_macro!()`.

cc rust-lang/rfcs#1913, rust-lang#38356

r? @jseyfried
cc @nrc
frewsxcv added a commit to frewsxcv/rust that referenced this pull request Mar 2, 2017
Implement function-like procedural macros ( `#[proc_macro]`)

Adds the `#[proc_macro]` attribute, which expects bare functions of the kind `fn(TokenStream) -> TokenStream`, which can be invoked like `my_macro!()`.

cc rust-lang/rfcs#1913, rust-lang#38356

r? @jseyfried
cc @nrc
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
T-lang Relevant to the language team, which will review and decide on the RFC.
Projects
None yet
Development

Successfully merging this pull request may close these issues.