Tracking issue for "Macros 1.1" (RFC #1681)

[nikomatsakis]

Tracking issue for rust-lang/rfcs#1681.

cc @alexcrichton

Stabilization TODO

Litmus tests:

• - Does diesel work? Tracking issue for "Macros 1.1" (RFC #1681) #35900 (comment)
• - Does serde work? https://users.rust-lang.org/t/serde-transitioning-to-macros-1-1/7437

Features:

• Crate name, currently is proc_macro
• Crate type, currently proc-macro
• The #[proc_macro_derive(Foo)] attribute
• Loading proc-macro crates with -L and #[macro_use] to load them
• shadowing is an error
• no hygiene
• passing a token stream for the entire struct and receiving it all back
• Cargo manifest attribute, currently proc-macro = true

Known bugs:

• - Panicking derive may have the wrong span - Second panicking custom derive has bad span #36935
• - token streams with mod foo fail - Custom derives: TokenStream::from_str fails if the string contains 'mod foo;' #36691
• - docs aren't published for proc_macro - Publish docs for proc_macro crate #38749
• - custom attributes for multiple modes is hard - Tracking issue for "Macros 1.1" (RFC #1681) #35900 (comment)
• - cargo test fails for proc macro libs - 'cargo test' with proc_macro_derive fails #37480
• - order still matters - Tracking issue for "Macros 1.1" (RFC #1681) #35900 (comment) (fxed by macros: expand #[derive]s after other attribute macros and improve intra-#[derive] ordering #37067)
• - Can't document rustc-macro crates - Can't document rustc-macro crates #36820 (fixed in rustdoc: Fix documenting rustc-macro crates #36847)
• - Cargo rebuilds too often - Dep info contains invalid file paths when rustc-macro is used #36625 (fixed in rustc: Use a special filename for macros 1.1 #36776)
• - Attributes generated by the compiler make custom derive authors' lives difficult - Tracking issue for "Macros 1.1" (RFC #1681) #35900 (comment)
  • - Change order of expansions to users-first - Tracking issue for "Macros 1.1" (RFC #1681) #35900 (comment) (fixed in rustc: Tweak expansion order of custom derive #36782)

Implementation TODO

• - Create a rustc_macro crate
  • - Have librustc_macro link to libsyntax. Depend on librustc_macro in librustc_driver
  • - Tag rustc_macro as unstable with our standard header.
  • - Only tag rustc_macro with #![crate_type = "rlib"], do not produce a dylib.
  • - Implement the API of rustc_macro using libsyntax's TokenStream internally
  • - tag rustc_macro with a TokenStream lang item so the compiler knows about it.
• - Add rustc_macro_derive attribute
  • - validate it's of the exact form foo(bar), no other arguments/formats
  • - verify it's only applied to functions
  • - verify it's only applied to functions in the root module
  • - verify the signature with the TokenStream lang item added above
  • - encode all function symbols with rustc_macro_derive into metadata along with the derive mode they're used for.
• - Add a rustc-macro crate type for other crates
  • - wire it up to produce a dylib
  • - ensure the dylib gets metadata
  • - ensure rustc-macro crates cannot be linked as dylibs
  • - ensure there are no reachable items other than those tagged with #[rustc_macro_derive]
  • - Add cfg(rustc_macro) as an unstable cfg directive, set it for the rustc-macro crate type
  • - ensure that rustc-macro crates link dynamically to libsytnax
• - Fill in #[macro_use] support for rustc-macro crates
  • - extend library loader to find rustc-macro crates separately from dylib/rlib tracked today when loding crates
  • - Parse metadata for rustc-macro crates to learn about symbol/derive mode pairings
  • - dlopen the dylib
  • - generate an error if any derive mode would shadow any other.
• - Add cargo integeration
  • - recognize rustc-macro similar to plugin = true
  • - pass --crate-type=rustc-macro when depending on it
  • - plumb the same host/target logic for rustc-macro crates as is present for plugin crates (e.g. always compile rustc-macro crates for hosts)
• - Tests
  • - smoke test loading a rustc-macro, dummy #[derive] trait
  • - name conflicts are an error
  • - compiling for the wrong architecture is an error
  • - cannot emit crate type rustc-macro and anything else (e.g. rustc-macro+dylib) is an error
  • - span information not being horrendous
  • - removing attributes from a struct, as well as fields
  • - adding impls next to a struct
  • - cross compilation looks for the host rustc-macro crate type
  • - don't load vanilla dylibs as rustc-macro crate types
  • - can't have public exports beyond macro derive functions in a rustc-macro crate
  • - derive macros must have required signature
  • - load two macro crates in one compilation

[alexcrichton]

I've updated the issue description with a checklist of what needs to be done. It's likely not exhaustive but it should get us 90% of the way there hopefully

[alexcrichton]

Ok, I'm going to look into this today and see how far I get.

[alexcrichton]

cc #35957, an implementation

[nrc]

From #35957: we should bikeshed the name of the librustc_macro crate some more. In particular, this is intended to be a long-lived crate which will have essentials for all macro authors in it, so limiting to rustc_macro (which in my mind, at least) is just about the 1.1 idea seems bad. I'd previously wanted libmacro for this, but since macro is a reserved word (and we might want it as a keyword in the future) that is impossible. @cgswords and I have been using libproc_macro, and I think that is not a bad name, although I am not 100% happy with it.

[eternaleye]

@nrc: Okay, my immediate thoughts on naming:

• extern crate macros; - short and sweet, but might be read as containing macros, rather than support code for writing them
• extern crate macro_runtime; - exactly what it says on the tin
• extern crate metarust; - write Rust, about Rust, for operating on Rust
• extern crate bikeshed; - with procedural macros, you can have whatever color of Rust you want!
• extern crate macrame; - sounds like "macro make[r]"; possibly better left for a future "nice" API over the raw library.

[aturon]

@nrc It seems like an important aspect of this question is the naming of our various macro styles over all -- in particular, if we go with libproc_macro, we're making a hard commitment to the "procedural macro" terminology. I don't have a strong opinion here, but I'm not sure if we've openly explored the space of terminology.

To be clear, are you thinking that today's macro_rules would simply be "macros", i.e. the default thing we mean by macros, while you have to qualify "procedural macros"? That seems like a pretty reasonable plan to me. And in that world, I'd argue that libproc_macro is better than libmacros.

[nrc]

My thinking here is that all macros are "macros", and when we need to make a distinction based on the implementation (the usage should be exactly the same for all kinds) we use "procedural macros" vs "macros by example". I would like to banish "syntax extension" and "compiler plugin" entirely (and one day re-use the latter for actual plugins).

But, yes, I strongly want to get behind the "procedural macro" terminology.

[aturon]

@nrc Makes good sense to me! While "macros by example" is a bit unwieldy, it's also a very evocative/intuitive term. My one worry about "procedural macro" is that "procedure" is not a thing in Rust. I wonder if there's a way of making the connection more direct. "Function macro" isn't quite right, but maybe gives you a sense of what I mean?

[nrc]

Yeah, it's not quite perfect, but given that it is a well-known/used term outside of Rust I think it is better than coining our own term. "Programmatic macro" is possible, but I prefer "procedural".

[eternaleye]

Perl's term for the closest equivalent it has to "procedural macros" is "source filters", which (especially with the shift from AST to tokens) is a pretty fitting description.

[mystor]

Perhaps 'Syntax transformers' or 'programmatic macros' would work well as names? I don't have a problem with procedural macros however.

[jimmycuadra]

"Procedural" is already what people call this, and I think it's clearly understood what it means, especially in contrast to "macros by example." I wouldn't worry about trying to find a different name.

[nikomatsakis]

I like the term "procedural macro" for regular use (or maybe "custom macro"). I particularly like using the word macro so that it's clear that they can (eventually...) be used in the same way as "regular macros". For this reason, I don't like "source filters" (I also expect a filter to just drop data, not transform it, though I know the term is used for both).

I am fine with either libproc_macro or libmacros. I sort of prefer the latter just because I don't love having _ in crate names when it can be readily avoided. =)

[nikomatsakis]

One question: do we ever expect to have "support routines" that could be used from non-procedural macros? I know of no such plans, but if we did, and we wanted them in the same crate, then libmacros would be a better name. =)

(I am thinking a bit about e.g., @dherman's comment from the eager-expansion RFC.)

[eternaleye]

@nikomatsakis: A related but subtly different question is a use case I brought up in rust-lang/rfcs#1561 (comment) - will we want procedural macros' implementation functions to be able to call other procedural macros' implementation functions?

I can easily see wanting to allow one custom derive to call another, and that would essentially make procedural macro definitions themselves capable of being used as such "support routines"

But yes, I think @dherman's gensym example is pretty compelling. Of course, if the answer to my question above is "yes", gensym is both a macro (which could be used by macros-by-example) and a support function (which could be used by procedural macros).

[dtolnay]

I have a cargo PR rust-lang/cargo#3064 which should check all the "cargo integration" boxes in the checklist.

[Ericson2314]

I left a comment on the cargo PR, but I think we want a different type of dependency, not just a different type of package. Firstly, I think this just is just better aestheticly and ergnomicly, but that's just my opinion. But I have two concrete reasons too.

• In a future with public and private deps, it's important to know that its to know that public deps of procedural deps and regular public deps need not coincide. E.g.

• In a future with inline procedural macros/quasi-quoting, any library can be used at compile time.

• will we want procedural macros' implementation functions to be able to call other procedural macros' implementation functions?

  As this shows the same crate could be a regular dep of another crate of procedural macros, or a macro dep of another crate.

[dtolnay]

Does serde work?

Yes https://github.com/serde-rs/serde/releases/tag/v0.8.6

(except for container attributes in some cases #36211)

[alexcrichton]

Awesome, thanks for the updates @dtolnay!

[White-Oak]

Are there docs on these macros? I suppose, the only example of using them is in serde, am I right?

[tomaka]

If the user writes #[derive(Foo)] #[foo_def = "definition.json"] struct MyStruct; the macro handler has no way to know what "the current directory" is and thus can't find definition.json.

This is by-design and thus wouldn't be easy to fix, and I guess it's too late to fix this anyway.

[eddyb]

You can go Span -> FileMap -> filename -> directory. All that's missing is accessing the information through proc_macro.

[tomaka]

You would also need to tell the compiler to add a dependency to definition.json so that the build is dirty if it gets modified.

[dtolnay]

The proc macro can use env::var("CARGO_MANIFEST_DIR") to get the directory containing Cargo.toml of the crate containing the macro invocation. Presumably foo_def is relative to that. See dtolnay/syn#70 (comment).

[est31]

@tomaka that can be done by mutating FileMap, e.g. this is how the include_str! macro does it.

[tomaka]

Presumably foo_def is relative to that.

I think it's not very intuitive to have to put the path relative to the Cargo.toml.

[tomaka]

that can be done by mutating FileMap, e.g. this is how the include_str! macro does it.

Yeah I know it can be done, it just can't be done with the current procedural macros API.

The parameter is an Item. It would be acceptable to add a method to retrieve a span from that Item, but adding a method to Item to ask the compiler to add a dependency would be a hack IMO.

[SimonSapin]

You can go Span -> FileMap -> filename -> directory.

Are these APIs (in particular FileMap) on a path to be stabilized?

[eddyb]

They don't have to be, in fact I wouldn't want to stabilize any of the internals. We can, instead, stabilize APIs that extract information about a Span (e.g. line, column, filename).

[alexreg]

I just got this error in a crate of mine. What's going on?

error: Cannot use `#![feature(proc_macro)]` and `#![feature(custom_attribute)] at the same time

[abonander]

@alexreg If you're using #[derive], it's stable now. You don't need #![feature(proc_macro)].

[jseyfried]

@alexreg
proc_macro_derives (macros 1.1) are now stable -- you can just remove #![feature(proc_macro)].

#[proc_macro_attribute] recently landed behind the #![feature(proc_macro)] feature gate; these are incompatible with #![feature(custom_attribute)]. #![feature(custom_attribute)] will be deprecated once the replacement lands (rust-lang/rfcs#1755).

[abonander]

@jseyfried I think we should change the tracking issue on proc_macro since this is closed and doesn't contain relevant information.

[alexreg]

Thanks guys. That makes sense.

[jseyfried]

@abonander Yeah, #![feature(proc_macro)] should definitely point to #38356 now -- I should have verified that when reviewing #38842. Could you submit a PR?