Macro naming and modularisation

[nrc]

This RFC proposes making macros a first-class citizen in the Rust module system.
Both macros by example (macro_rules macros) and procedural macros (aka syntax
extensions) would use the same naming and modularisation scheme as other items
in Rust.

rendered

[brendanzab]

This would be very useful for library developers, especially if we could reexport them as well! For example I would love this for my approx crate. Not sure about any subtle edge cases in this proposal though.

[oli-obk]

We could simply lint on #[macro_use] and #[macro_export]. This would never show up in dependencies, but only in the crate that's currently compiled

[ahicks92]

A question: does this mean I can put a macro in an impl block?
At the moment, we have MyType::new, the equivalent of a constructor. For data structures that wish to be initialized with lists of items, it might also be helpful to do MyType::new!, i.e. Vec::new![1, 2, 3, 4, 5] as opposed to the gloval vec! macro.
If not, it should be noted that this would solve the most common case for which I could see a C++-style varargs or initializer list being useful while avoiding the pollution problem that we have currently.

[nrc]

@camlorn No. Name resolution happens before type information is collected, so that wouldn't necessarily follow from this RFC. This RFC does not preclude doing that in the future, but I don't see a nice way to implement it in the general case.

[xeno-by]

Could you provide some hints on how you plan to implement this?

[nrc]

Procedural macros and any items marked with #[cfg(macro)] would be split off and compiled into one crate, everything else compiled into another. The first crate would be linked in to the compiler while it compiles the second (or equivalently, called using IPC).

[xeno-by]

So macros will only be able to call functions marked with #[cfg(macro)]?

[eddyb]

That seems strange, #[cfg]-gating can only remove items with #[cfg(P)] where the cfg predicate P doesn't hold.

[solson]

@nrc The Rendered link in the original post links to the original commit (https://github.com/nrc/rfcs/blob/573bd83aaac0896842f136262458c3b3bf671cd7/text/0000-macro-naming.md) rather than the most up-to-date version (https://github.com/nrc/rfcs/blob/macro-naming/text/0000-macro-naming.md).

[withoutboats]

In the original blog series, it was suggested that macros imported using this system would be imported using a new symbol. I don't see that in this RFC, so I take it that this will change how macro_rules! macros can be imported. Are there any backwards compatibility issues connected with this change?

[nrc]

In the original blog series, it was suggested that macros imported using this system would be imported using a new symbol. I don't see that in this RFC

This is #1584.

[paulstansifer]

I don't think this is a good idea, because it won't play nice with macro-defining macros. For example, the following sort of thing should (and currently does) work:

  macro_rules! macro_defining { 
    ($name:ident) => { macro_rules! $name { () => (println!("Yay!");) } }
  }

  macro_defining!( say_yay );

  say_yay!();

...but it only works if the macros are expanded in the order they are written, and there's no reasonable static analysis that can tell us that fact.

[paulstansifer]

Regarding the issue of ! in use statements: since macro! isn't nailed down yet, we can achieve total consistency:

macro! {
  my_macro!(...) => {...}
}

and

use some_mod::my_macro!;

I think that this would be perfect if it weren't for attribute-style macros. Even so, I think that I prefer to use ! as if it were really part of the name for non-attribute-style macros:

use some_mod::{my_normal_macro!, my_attribute_macro, my_ordinary_function};

I feel like this is what I would try if I didn't know what the actual rule was.

[solson]

@paulstansifer There has been a lot of discussion on that topic, but I guess I'll point to #1561 (comment) in particular.

It doesn't seem worthwhile to use ! here, and it's more consistent to keep use syntax uniformly unadorned.

[eddyb]

I was under the impression definitions and uses would be orthogonal even if we had two systems for each, but @nrc pointed out on IRC that is not the case as currently proposed.

To which I make the counter-suggestion: allow the usage of #[macro_use] or use, regardless of how the macro was defined or even exported, at least across crates (as long as the macro is visible at the top-level of the crate from which the macro is being imported, in thr case of #[macro_use]).
At the same time, this provides 2 benefits:

• allow crate authors to update without breaking their old users
• allow crate users to upgrade without changing old dependencies

[nikomatsakis]

Hear ye, hear ye! This RFC is now entering final comment period. In the most recent @rust-lang/lang meeting, @nrc and I (the only two in attendance) felt inclined to accept. In particular, we're definitely in favor of the high-level aims of this RFC (controlling macros via ordinary use statements).

The major points of interest in the thread so far have been:

• whether to place macros in their own namespace (the RFC does)
• whether to import macros using a trailing ! (use foo::bar vs use foo::bar!). We two felt that use foo::bar was better, since the ! is more properly considered a "use macro operator" and not part of the macro name. As evidence for this position, consider that one can write higher-order macros that operate like macro_rules! foo { ($x:ident, $y:tt) => { $x!($y) } } (invoked like foo!(bar, baz), which is roughly equivalent to bar!(baz)). See this example on play. There is some good discussion here.
• @paulstansifer's contention that not rely on macro-ordering will break macro-defining-macros; this is worth discussing more (I believe that not to be true, but there are some differences that emerge versus systems like Racket, which have a more mutable view of the universe).

(There may be some implementation details to be hashed out.)

[nikomatsakis]

@paulstansifer

I don't think this is a good idea, because it won't play nice with macro-defining macros

This is definitely an important point, thanks for raising it. In fact, I think the system as we've envisioned it plays great with macro-defining macros, but there are some patterns in Racket that will not work. Let's start with your example:

macro_rules! macro_defining { 
    ($name:ident) => { macro_rules! $name { () => (println!("Yay!");) } }
}
macro_defining!( say_yay );
say_yay!();

This interacts with the overall name resolution algorithm defined in #1560. The idea is roughly that expansion and name resolution is a process that occurs in rounds. In this case, in the first round, the path say_yay! will be "unresolved" -- because there is no definition for it. However, macro_defining! will resolve and expand. In the next round, then, say_yay will resolve. Errors are only reported if we have expanded all macros and still have paths that cannot be resolved at the end of the process.

One danger here is that expanding a macro may cause a path that resolved in the past to resolve differently. Mostly this isn't possible, because expanding a macro can only add items, and duplicate items cause errors. However, one case where it can cause trouble is around globs and shadowing -- if a path relies on item X that is later shadowed by another item X produced by a macro, then it resolves differently. To counter this, we make it an error (in #1560) for a macro to expand to an item X that shadows an item in an outer scope, at least if that item was used in a macro expansion path. (See @jseyfried's comment for more details.)

Personally I find the idea of item ordering being significant in Rust to be very unappealing. It is strikingly different from how most things in Rust work, and it particularly doesn't mix well with use statements in my mind. (Perhaps the ordering could be some sort of DAG derived from the use statements, but my efforts to find such a system that works out well mostly failed.)

But this does make it more difficult to have "comunicating" macros, which I understand in Racket at least often rely on expansion order. An example that I was given by @samth where this could break Racket patterns of macro interop was something about pattern matching. I think the idea was that a class! macro might generate some metadata that is later used by a match! macro (in Racket, I mean), and that this introduces an ordering dependency. However, he also added that in practice macros like match! would generally try to "reschedule" themselves to run late in the expansion process so that the relative ordering of a class declaration and a match wouldn't be important. (I don't remember more details than that.)

I think a potential example of who this could come into play in Rust might be that, e.g., a macro might want to access data from a type definition. (Perhaps one would be expected e.g. to annotate the type definition in one way, and this would enable uses elsewhere that "communicate" with that annotation to extract data about the types of the fields, or something.) This is not well supported in the current system, which is targeted at more "self-contained" macros.

I'm not sure what's a "general" fix here but I think a lot will depend on the details. For example, if the "uses" that wish to communicate with the definition are inside of fns or other code blocks, I expect we can do things to ensure that all module-levels macros and items are expanded first (given that names defined in fns can't be used from outside that fn).

I can also imagine patterns where a module is decorated (#[foo] mod { ... }) -- this would then allow code in the module to reference data generated by the #[foo] macro. This works just by default because the code that will reference what #[foo] generates is ultimately governed by foo and results from the expansion of foo itself.

Finally, we might develop somewhat more ad-hoc solutions, similar to how Racket macros might "reschedule" themselves to make themselves order independent, where a macro can ask to be deferred from executing if it finds that other macros it might depend on are not yet expanded.

In short, I'd rather tackle this in a declarative way than relying on the "AST ordering", which I don't think is a natural concept in Rust.

(As one last point, I'd sort of like to get rid of explicit mod declarations, at least for private modules, and instead infer their presence from the file-system -- but if we did so, it would interact very poorly with having a defined AST ordering.)

[ahicks92]

Can we get a macro_namespaced attribute added to current macros which would allow use of this feature now?

As I read it, this is applying to a hypothetical future in which we rewrite macros by example to use a new macro! keyword. But as far as I know, there isn't much of a timeline.

Specifically:

• A macro with macro_namespaced requires explicit importing and reference per this RFC to be used.
• All macros that choose not to use it can be used as they are now.
• All macros that choose not to use it are treated as being in the root of their crate and may be used in the new style as well as the old.

I would like to see this feature before the macro redesign is finished, and this seems like a good compromise that allows for code transition. The disadvantage is that I believe the old macro_rules stuff is on its way out, and obviously this adds something else to it.

The name is off the top of my head. I'm not attached to it.

[withoutboats]

Maybe I missed it, but the RFC doesn't seem say anything about redefining / shadowing rules for macros in regard to other macros. Currently, the macro rules for this work very differently from the rules for functions. Currently, you can redefine macros whenever you want, and whichever definition is most recent in the source at the expansion site is the definition that is used.

If macros use the same import system as other symbols, I would expect their shadowing behavior to be the same. That is, you can't provide two definitions in the same module or import and define, but you can shadow prelude macros and you can shadow macros by providing a definition inside the body of a function.

(As one last point, I'd sort of like to get rid of explicit mod declarations, at least for private modules, and instead infer their presence from the file-system -- but if we did so, it would interact very poorly with having a defined AST ordering.)

(This is totally offtopic but this seems like a very good idea in the long term which would make Rust's module story more grokkable to users coming from other languages which work this way)

Also this PR needs the final-comment-period label. :-)

[strega-nil]

Agreed with @withoutboats about shadowing.

use foo::bar is much better than use foo::bar!

placing macros into their own namespace is also a good idea; it'd be weird to have macros be in the same namespace as types and values.

[arielb1]

(As one last point, I'd sort of like to get rid of explicit mod declarations, at least for private modules, and instead infer their presence from the file-system -- but if we did so, it would interact very poorly with having a defined AST ordering.)

But I like explicit mod declarations. The "every crate is 1 file" model is so liberating.

[nikomatsakis]

Can we get a macro_namespaced attribute added to current macros which would allow use of this feature now?

That's a good question. I would hope we could have some way to use this with old-school macros, whether that require some sort of opt-in, or maybe we can make it work by default without breakage.

[nikomatsakis]

If macros use the same import system as other symbols, I would expect their shadowing behavior to be the same.

I'm a bit confused by this comment -- are you saying that this is why we can't easily use these new rules for existing macros? Or are you saying that you believe the newer rules in this RFC would cause macros to shadow in different ways than other names that are brought into scope via use? (Or are you referring to the errors that can result if you generate an item via a macro?)

EDIT: Re-reading, I see that you're just saying that the shadowing rules are not explicit enough in the RFC text. I think the answer is that they would follow the same rules as any other items, because we will literally be resolved macro paths using the same lookup rules we use for any other path.

[nikomatsakis]

We discussed this in the @rust-lang/lang meeting and decided to leave it open for FCP a little longer. For one thing, we wanted to incorporate various bits of the discussion in to the RFC text. Nonetheless, I think we're all feeling generally positive about this direction.

[paulstansifer]

The multi-round approach sounds tricky, but it's a really cool idea, and I have to say that I like the added consistency between macro definitions and normal items. I don't think that macros communicating via side-effects is a great idea in the first place (though, to be honest, I've never seen it up close, so I may not be the best judge); it seems like macro expansion ought to be side-effect-free. It feels like the overall effect is to move macro expansion from feeling eager and imperative to feeling lazy and declarative. Seeing as the outside-in macro expansion order is already sort of lazy-feeling, this might give macro expansion a more consistent overall "feel".

[nikomatsakis]

Huzzah! The @rust-lang/lang team has decided to accept this RFC.

[nikomatsakis]

Tracking issue: rust-lang/rust#35896

If you'd like to keep following the development of this feature, please subscribe to that issue, thanks! :)