Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking issue: declarative macros 2.0 #39412

Open
11 of 19 tasks
nrc opened this issue Jan 30, 2017 · 78 comments
Open
11 of 19 tasks

Tracking issue: declarative macros 2.0 #39412

nrc opened this issue Jan 30, 2017 · 78 comments
Labels
A-macros Area: All kinds of macros (custom derive, macro_rules!, proc macros, ..) B-RFC-approved Blocker: Approved by a merged RFC but not yet implemented. B-unstable Blocker: Implemented in the nightly compiler and unstable. C-tracking-issue Category: An issue tracking the progress of sth. like the implementation of an RFC F-decl_macro `#![feature(decl_macro)]` S-tracking-design-concerns Status: There are blocking design concerns. T-lang Relevant to the language team, which will review and decide on the PR/issue.

Comments

@nrc
Copy link
Member

nrc commented Jan 30, 2017

Tracking issue for declarative macros 2.0 (aka macro aka decl_macro aka macros-by-example).

RFC: https://github.com/rust-lang/rfcs/blob/master/text/1584-macros.md

RFC PR: rust-lang/rfcs#1584

cc @rust-lang/compiler


About tracking issues

Tracking issues are used to record the overall progress of implementation.
They are also used as hubs connecting to other relevant issues, e.g., bugs or open design questions.
A tracking issue is however not meant for large scale discussion, questions, or bug reports about a feature.
Instead, open a dedicated issue for the specific matter and add the relevant feature gate label.
Discussion comments will get marked as off-topic or deleted.
Repeated discussions on the tracking issue may lead to the tracking issue getting locked.

Tasks

Potentially blocking issues:

@nrc nrc added A-macros Area: All kinds of macros (custom derive, macro_rules!, proc macros, ..) B-RFC-approved Blocker: Approved by a merged RFC but not yet implemented. B-unstable Blocker: Implemented in the nightly compiler and unstable. T-lang Relevant to the language team, which will review and decide on the PR/issue. labels Jan 30, 2017
@CryZe
Copy link
Contributor

CryZe commented Jan 30, 2017

@nrc Typo in Issue Name: "issuse"

@nikomatsakis nikomatsakis changed the title Tracking issuse: declarative macros 2.0 Tracking issue: declarative macros 2.0 Jan 30, 2017
@jseyfried
Copy link
Contributor

jseyfried commented Feb 7, 2017

Tasks

(dtolnay edit: moved the checklist up to the OP)

@nrc
Copy link
Member Author

nrc commented Feb 7, 2017

We need to RFC a whole bunch of stuff here. In particular, I would like to propose some new syntax for declaring macros and we should RFC the changes to matchers.

@tikue
Copy link
Contributor

tikue commented Mar 10, 2017

Can the hygiene RFC mention pattern hygiene? This in particular scares me:

// Unsuspecting user's code
#[allow(non_camel_case_types)]
struct i(i64);

macro_rules! ignorant_macro {
    () => {
        let i = 0;
        println!("{}", i);
    };
}

fn main() {
    // oh no!
    ignorant_macro!();
}

@jseyfried
Copy link
Contributor

jseyfried commented Mar 13, 2017

@tikue I'm not sure patterns need special treatment with respect to hygiene.

For example, on the hygiene prototype,

#[allow(non_camel_case_types)]
struct i(i64);

macro ignorant_macro() {
    let i = 0; // ERROR: let bindings cannot shadow tuple structs
    println!("{}", i); 
}

fn main() {
    ignorant_macro!(); // NOTE: in this macro invocation
}

This makes sense to me since let i = 0; is shadowing a tuple struct.
In particular, if let i = 0; were removed then the following use of i would resolve to the tuple struct, no matter where ignorant_macro is used.

Note the symmetry to this example:

#[allow(non_camel_case_types)]
struct i(i64);

fn ignorant_fn() {
    let i = 0; // ERROR: let bindings cannot shadow tuple structs
    println!("{}", i); 
}

If the tuple struct i isn't in scope at macro ignorant_macro() { ... }, then the let i = 0; does not shadow it and there is no error. For example, the following compiles on the hygiene prototype:

mod foo {
    pub macro ignorant_macro() {
        let i = 0;
        println!("{}", i); // Without `let i = 0;`, there would be no `i` in scope here.
    }
}

// Unsuspecting user's code
#[allow(non_camel_case_types)]
struct i(i64);

fn main() {
    foo::ignorant_macro!();
}

@Mark-Simulacrum Mark-Simulacrum added the C-tracking-issue Category: An issue tracking the progress of sth. like the implementation of an RFC label Jul 22, 2017
@alexreg
Copy link
Contributor

alexreg commented Aug 17, 2017

Has there been any progress on Macros 2.0 lately?

@mark-i-m
Copy link
Member

mark-i-m commented Jan 30, 2018

Note for those who haven't seen yet: macros 2.0 is apparently slated to be stable later this year, according to the proposed roadmap (rust-lang/rfcs#2314)...

On the one hand, that's pretty exciting 🎉!

On the other hand, I was really surprised that the feature is so close to being done and so little is known about it by the broader community... The RFC is really vague. The unstable book only has a link to this issue. This issue(s) in the issue tracker mostly have detailed technical discussions. And I can't find that much info anywhere about what's changed or how stuff works.

I don't mean to complain, and I really truly appreciate all the hard work by those implementing, but I would also appreciate more transparency on this.

@alexreg
Copy link
Contributor

alexreg commented Jan 30, 2018

@marcbowes Yeah, I'm kind of worried too. It seems like there's quite a lot of work left for stabilisation this year. I offered to work on opt-out hygiene for identifiers myself, but have received no response yet... Some greater transparency would be nice, as you say.

@petrochenkov
Copy link
Contributor

Macros 2.0 are not even in the RFC stage yet - the @jseyfried's RFC linked in the issue was never submitted, there's only very high level RFC 1584.
The implementation is purely experimental and mostly unspecified, and large part of it (not related to hygiene) is reused from macro_rules! without fixing problems that macros 2.0 were supposed to fix.

Some greater transparency would be nice

The problem is that no work happen right now, so there's nothing to reveal :(
@nrc is busy, @jseyfried is busy, I worked on macros 2.0 a bit and would really like to dig into them more, but I'm busy too, unfortunately.
We need someone to adopt the feature, support and extend it, and gain expert-level knowledge of it, otherwise we will end up breaking hygiene and syntax details left and right after stabilization.

@alexreg
Copy link
Contributor

alexreg commented Jan 30, 2018

@petrochenkov Ah, fair enough. I mean, that's a shame, but it makes sense at least. I guess this is an open call to anyone who might be able to take ownership of this feature, since no one comes to mind? I could still have a go at a little sub-feature, but I'm certainly in no position to take ownership of this.

@petrochenkov
Copy link
Contributor

petrochenkov commented Jan 30, 2018

@alexreg

I could still have a go at a little sub-feature

Yes, please do!
Hygiene opt-out is one of the primary missing parts and implementing it would be useful in any case.

IIRC, two questions will need to be decided on during implementation:

@alexreg
Copy link
Contributor

alexreg commented Jan 30, 2018

Syntax for "unhygienic" identifiers (@jseyfried tentatively suggested #ident in #40847)

Yeah, this was the plan. :-)

What exactly hygienic context the identifier introduced with #ident in a macro m will have - context of m's invocation? context after expanding all macros ("no hygiene")? something else?

What's your inclination? I'm leaning towards the context of m's invocation, but curious to hear your thoughts...

@clarfonthey
Copy link
Contributor

Presumably it will be stabilised after all of the requisite tasks in the issue description are completed.

@cybersoulK
Copy link

cybersoulK commented Aug 9, 2023

I might not fully grasp the nuances of macros 2.0, but these are the benefits I anticipate:

1 - I should be able to use crates, types, and functions that I define in my library directly without needing $crate (suggesting deprecation of $crate).
2 - Every item created within the macro should expand; after all, that's the core purpose of macros.

This reasoning is sound:

If you desire certain types or functions to remain private, simply define them outside the macro where they remain accessible to the macro.
If an item is dynamic and changes based on token input, it still needs to be expanded into the scope. Given that it's a distinct type, it shouldn't just disappear.
The types or variables introduced during expansion shouldn't conflict with tokens, even if their names are identical. Think of library-defined code as being in red, and tokens in blue; they are distinct entities.
The user decides where the macro expands. They can enclose it using a mod { macro!() } or restrict its scope using a block { macro!() }.
At its core, macros operate on this principle: Library code + User code (Tokens) = Output Expansion.
I'm concerned that we might be overcomplicating the concept of hygiene.

@cybersoulK
Copy link

cybersoulK commented Aug 9, 2023

From my previous points, I believe the confusion we're facing stems from how to handle situations where Library code intertwines with User code.
It's naturally expected for them to merge within macros, since, by definition, a macro represents the library code's blueprint being replicated into the user's domain. Perhaps there's a need for enhanced tools to manage instances when two textual terms conflict, rather than sidestepping the issue and introducing myriad limitations to macros.


yea, i don't see why this is hard to do, if the macro has a chance of word collision.

mod macro_mod {
macro!();
}

pub use macro_mod::Type as NewNamedType;

i would not be against if the macro 2.0 used mod by default when expanding in open scopes, such as

use crate::macro;
macro!();
pub use macro::Type as NewNamedType;

use crate::macro as macro_renamed;
macro_renamed!();
pub use macro_renamed::*;

//automatically created macro mod and macro_renamed mod.

But then there must be a differentiation of macros that contain function calls and variable assignments, and should be expanded directly, such as:

match ... {
   A(name) => macro!(name),
}

@jjpe
Copy link

jjpe commented Aug 9, 2023

@cybersoulK do everyone here a favor please, and don't multipost. Everyone who is subscribed to this gets a separate email about each post, and the signal:noise ratio is dropping real fast ATM.

To be clear: posting is fine, but maybe don't drive people up the wall with constant useless notifications.

@dengyunsheng250
Copy link

Has there been any progress on Macros 2.0 lately?

@jhpratt
Copy link
Member

jhpratt commented Aug 22, 2023

No, but there will hopefully be a general increase in knowledge in what the issues are, as the proposal for a macros working group has been accepted. The WG will be created shortly!

@tgross35
Copy link
Contributor

tgross35 commented Sep 13, 2023

It could be interesting to allow a way to reverse parsing to match from right to left. This can be faked using an accumulator and recursively popping tokens until you get a match on a single one, but this is a pretty messy pattern. A more simple way to create a reverse TT muncher would make it much easier to apply precedence to contents, such as when evaluating math.

// Works
macro first {
    ($first:tt $($rest:tt)*) => { println!("first item is {}", $first) };
}

// Fails to compile (multiple parsing options)
macro last {
    #[reverse] // maybe this could make it work
    ($($rest:tt)* $last:tt) => { println!("last item is {}", $last) }
}

Edit: an alternative to this would just be adjusting the rules for multiple parsing such that it tries right to left if it gets stuck. I think this would accomplish the same thing without needing an annotation.

@clarfonthey
Copy link
Contributor

The main issue with RTL parsing is that, at least for now, parsing doesn't actually store the tokens being returned in a way that lets you iterate backwards. You could allow this and do it without allocating if you permit parsing the tokens twice (once to find the end of the macro invocation, the second time to do the actual parsing), but it's still a performance penalty to allow parsing from the other direction.

Of course, the usual way, recursively expanding an intermediate macro in reverse order, is probably slower than offering a dedicated method that allows this. If this were done, it would be nice to also have proc_macro::TokenStream implement DoubleEndedIterator to reflect that, and allow proc macros to also take advantage of that.

@Ygg01
Copy link

Ygg01 commented Sep 15, 2023

Here is to hoping two big issues are getting solved (hygiene bending and nesting macros).

I tried solving macro hygiene but one pre-requisite was learning how other languages do it. Learning Racket is where I lost my motivation.

@petrochenkov
Copy link
Contributor

Here is to hoping two big issues are getting solved (hygiene bending and nesting macros).

I tried solving macro hygiene but one pre-requisite was learning how other languages do it. Learning Racket is where I lost my motivation.

The issue about hygiene is "Tracking issue for Span::def_site()", this issue is blocked on it.
In general, declarative macros are a syntactic sugar for procedural macros, so the "procedural macros 2.0" need to be implemented first.

Racket needs to be learned just enough to understand https://users.cs.utah.edu/plt/scope-sets/ in detail, to be able to possibly tweak it to match Rust realities.

@lolbinarycat
Copy link
Contributor

one usecase of macros where rust does worse than C is local unhyginic macros used in the definition of a single array or function.

some way to bind specific variables to to those from the outer scope, or to disable macro hygine for certain identifies, would be appreciated.

perhaps i need to call a function 10 times, and each time i need to pass 7 different arguments, 6 of which are always the same local variables, and only the 7th changes. C could easily manage this with a define and undef, but in rust, you would have to pass all the local variables to each macro call, which wouldn't actually simplify the code at all.

@felix91gr
Copy link
Contributor

which wouldn't actually simplify the code at all.

That is true for this version of "simplify":

Simpler code: shorter code.

But it is not true for the following version of "simplify":

Simpler code: code that is easier to understand and maintain.

I don't think locally disabling hygiene helps at all, if what we seek is simpler code in the second sense.

@ifsheldon
Copy link

one usecase of macros where rust does worse than C is local unhyginic macros used in the definition of a single array or function.

I think "worse" here is generally better. I've read a pretty bad C codebase once, which is full of unhygine macros that means a lot of sloppy auto copy-and-paste. IDEs could not even help me to analyze the code. I really appreciate Rust didn't just choose the simple way to do macros.

perhaps i need to call a function 10 times, and each time i need to pass 7 different arguments, 6 of which are always the same local variables, and only the 7th changes.

Make a closure please.

@tmccombs
Copy link
Contributor

Make a closure please.

A closure can't do everything a macro can. For example it can't return from the parent function.

@mayabyte
Copy link

perhaps i need to call a function 10 times, and each time i need to pass 7 different arguments, 6 of which are always the same local variables, and only the 7th changes.

I think what you're describing is partial application which is something that can already be achieved with existing crates. Combining this with the Try operator could achieve the early-return functionality described above as well.

@lolbinarycat
Copy link
Contributor

it turns out the existing macro_rules! macros can already access identifiers from their parent scope, so this type of local macro actually already works, for when you need to do something that partial application can't do.

@oli-obk
Copy link
Contributor

oli-obk commented May 20, 2024

Tracking issues are not intended as places for discussions. This issue already has over 100 comments and lots of subscribers. Please open separate issues and link them, instead of having the discussion here.

@safinaskar

This comment has been minimized.

@tgross35

This comment has been minimized.

@safinaskar

This comment has been minimized.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-macros Area: All kinds of macros (custom derive, macro_rules!, proc macros, ..) B-RFC-approved Blocker: Approved by a merged RFC but not yet implemented. B-unstable Blocker: Implemented in the nightly compiler and unstable. C-tracking-issue Category: An issue tracking the progress of sth. like the implementation of an RFC F-decl_macro `#![feature(decl_macro)]` S-tracking-design-concerns Status: There are blocking design concerns. T-lang Relevant to the language team, which will review and decide on the PR/issue.
Projects
None yet
Development

No branches or pull requests