Expand items before their derives #48465

+        let item = item.map_attrs(|mut attrs| {
+            attrs.retain(|a| a.path != "structural_match" && a.path != "rustc_copy_clone_marker");
+            attrs
+        });


@abonander
Do you know why these attributes are filtered away here?
They are normally produced by builtin derives, but I'm not sure how they affect custom derives and why they need to be filtered away before applying a custom derive.

@jseyfried added this in the original PR. Naive guess, it's avoiding exposing implementation details to the custom derive? Because these would be retained in the actual item since custom derives don't re-emit the item they're applied to.

custom derives don't re-emit the item they're applied to

Aha, this must be the key observation. Everything looks reasonable then.

petrochenkov · 2018-03-04T18:04:21Z

src/libsyntax/ext/expand.rs

-    fn collect_invocations(&mut self, expansion: Expansion, derives: &[Mark])
-                           -> (Expansion, Vec<Invocation>) {
-        let result = {
+    fn collect_invocations(&mut self,


@abonander
Did you figure out what happens here and above in fn expand?
I can't understand it just by reading the code, but it usually becomes more clear after actively doing some refactoring or modifications.

I'm not sure what you mean by "figure out what happens here". It seems to do what it says on the tin; it accumulates bang/attribute macro invocations and then expand expands them. It has to be done in a loop because macro expansions can produce more invocations.

petrochenkov · 2018-03-04T18:06:52Z

src/libsyntax/ext/expand.rs

    monotonic: bool, // c.f. `cx.monotonic_expander()`
 }

+/*


Nit: comment style, // please.

petrochenkov · 2018-03-04T18:11:28Z

src/libsyntax_ext/deriving/generic/mod.rs

-    struct Visitor<'a> {
+    struct Visitor<'a, 'b: 'a> {
+        cx: &'a ExtCtxt<'b>,
+        span: Span,


The original PR removed these fields and fn visit_mac below.
Are they needed again for some reason?

petrochenkov · 2018-03-04T18:14:09Z

src/librustc_resolve/macros.rs

-            for &(derive, _) in derives {
-                unresolved.insert(derive);
-                self.invocations.insert(derive, invocation);
-            }


Hmm, this commit also reverts some changes done in the original PR.

This and the other nonsensical changes are probably due to a naive rebase where I just applied all nonconflicting changes from both sides. I'll have to go through and fix these or redo the rebase.

petrochenkov · 2018-03-04T18:20:22Z

Ok, r=me once Travis is green and all the rebase issues are cleaned up (ideally by doing a "less naive" rebase).

EDIT: Also squashing non-@jseyfried commits would be nice.

abonander · 2018-03-19T07:55:07Z

@petrochenkov Since I don't completely understand everything that this code is doing, I'm slightly concerned about derives that declare custom attributes: it seems that this should break it since we would always end up looking at the attributes first and throw an error since they don't resolve to anything. However, the following tests using custom attributes are passing:

Do you think these are enough of a smoke test that we don't need a separate test to ensure this hasn't broken custom attributes for derives?

@jseyfried could you weigh in on this since it's mostly your code?

aidanhs · 2018-03-20T01:35:03Z

Just to give an update, crater run will start in ~4 days. Sorry :( This PR got pushed down because a more time-critical one cropped up.

petrochenkov · 2018-03-20T21:37:10Z

@abonander

I'm slightly concerned about derives that declare custom attributes: it seems that this should break it since we would always end up looking at the attributes first and throw an error since they don't resolve to anything

I'm not sure what is this about exactly.
The linked tests don't contain attributes declared by derives? (I.e. produced by expansion of derive.)
IIRC, attributes are processed in the same order they are written? (Except for maybe cfg?) I.e. #[foo] is processed after #[derive(Foo)] in the linked example. Does this PR change this behavior?

abonander · 2018-03-20T21:47:26Z

The linked tests don't contain attributes declared by derives? (I.e. produced by expansion of derive.)

I thought attributes declared by derives are meant to be applied by the user and consumed by the derive, i.e. removed after the derive completes. This is how Serde uses them anyway: https://serde.rs/attributes.html

This PR, at least nominally, expands derives last on an item, processing other macros first: https://github.com/rust-lang/rust/pull/48465/files#diff-54079e90040377e52f969c857e3f558aR21

If we expand the item itself first before its derives, the expander would look at these custom attributes and find that they don't resolve to anything and throw an error. However, the tests I linked are passing which suggests this has already been accounted for.

petrochenkov · 2018-03-20T21:53:15Z

I suspect that the expander classifies the attributes first (into builtin, unknown, whitelisted, macro-attributes of various kinds of legacy-ness) and expands only those that are actually macros.

keeperofdakeys · 2018-03-20T22:55:55Z

The ProcMacroDerive::expand function marks all matching attributes as used and known when expanding a proc_macro_derive, so the compiler ignores them. They were once removed, but that prevented multiple derives using the attribute, and meant the derive had to return a modified item.

IIRC this works because errors regarding macro resolution only become a hard error once everything has expanded.

This change doesn't make derives happen last, it makes bang macros happen first (so called partial expansions).

abonander · 2018-03-20T22:57:54Z

Mmm, good assessment. I've only recently started getting back into macro expansion stuff so a lot of these changes went over my head.

aidanhs · 2018-03-26T12:13:57Z

Crater run started

nikomatsakis · 2018-03-26T14:58:27Z

This came up again today when discussing procedural macros (e.g., Macros 1.2). I'm still pretty hesitant to make changes here -- it's not clear that expanding before derive is what we want to do.

jseyfried · 2018-03-26T18:06:48Z

(Re)reviewed, LGTM. Were there any other questions about the code not answered by #48465 (comment) or #48465 (comment)? (thanks @keeperofdakeys!)

jseyfried · 2018-03-26T18:09:40Z

@nikomatsakis Has @nrc changed his mind?

The main motivation for this was that we already "expand" #[cfg] and #[cfg_attr] before derive, and this is widely relied upon today. If we expand everything, then we can think of cfg and cfg_attr as just other attribute macros. This is useful e.g. so that people can implement their own customized my_cfg attribute macro that behaves exactly like cfg where desired.

nrc · 2018-03-27T19:30:31Z

Has @nrc changed his mind?

Yes. Sorry, I did not at all understand what is going on here. I don't think we can land this because changing expansion order is a breaking change, but also because the more complex the expansion order, the harder it is to understand what is going on. The fact that cfgs are evaluated out of order with respect to macros is an unfortunate historical artefact and shouldn't be considered precedent here.

The trouble I think is that one can create examples which benefit from either expansion order. Given that, I think we must go for the most predictable and consistent behaviour, which I believe is the current behaviour.

Apologies that I didn't get to this conclusion earlier.

abonander · 2018-03-27T19:33:37Z

@nrc What about making the expansion order opt-in? We could apply some derives pre-expansion and some derives post-expansion.

jseyfried · 2018-03-27T20:40:38Z

@nrc Ok.

To be clear, cfgs are only evaluated out of order with respect to derive macros, not any other kind.

I don't think we can land this because changing expansion order is a breaking change

I doubt this is breaking in practice, but yeah back-compat could complicate things...

The fact that cfgs are evaluated out of order with respect to macros is an unfortunate historical artefact and shouldn't be considered precedent here.

I think it's worth noting though that just about every major derive macro today needs cfg to be expanded first. That is, if the historical artifact had not existed, just about every derive macro would need to opt in to (at least) cfg getting expanded first (via expand: Tokenstream -> Tokenstream or some other mechanism) to support people deriving things on structs with conditionally compiled fields.

The trouble I think is that one can create examples which benefit from either expansion order

Can you think of examples of derives that don't want their input fully expanded? These seem like a far smaller class, especially if we assume that attribute macros will eventually be allowed on struct fields.

Given that, I think we must go for the most predictable and consistent behaviour

I think most predictable/consistent would be either everything is expanded first, or the macro sees the original, raw, unmodified Tokenstream from the source code. Halfway seems weird.

I like @abonander's idea of allowing the macro author to request either unexpanded or fully expanded input. If derive back-compat is an issue, not requesting either could default to today's behavior.

abonander · 2018-03-27T21:05:59Z

Something like

#[proc_macro_derive(MyTrait, expand_input = "true")]

Or do we want to provide it as a general mechanism in libproc_macro like @jseyfried suggested with pub fn expand(TokenStream) -> TokenStream?

jseyfried · 2018-03-27T21:55:20Z

Just providing expand: TokenStream -> TokenStream wouldn't be enough to opt out, which presumably we'd want since at least cfg will continue to be expanded first for derives by default.

abonander · 2018-03-27T21:59:51Z

I meant for that to be the way to opt-in to input expansion, by programmatically asking for it:

#[proc_macro_derive(MyTrait)]
pub fn my_trait_derive(input: TokenStream) -> TokenStream {
    let expanded = proc_macro::expand(input);
    // ...
}

nrc · 2018-03-28T15:18:20Z

What about making the expansion order opt-in?

So, for proper proc macros my preferred way to opt-in is to let macros eagerly expand macros in their body by adding expand functionality to the API available to proc macros. This also helps custom derive in the long term if you rewrite your custom derive to be token-based rather than string-based. However, this is kind of a long way off. Personally I'm happy to wait. Custom derive was always meant to be an 80/20 solution and very rough and ready. We could have some opt-in as part of the custom derive attribute, but the motivation would have to be very strong IMO.

Can you think of examples of derives that don't want their input fully expanded?

macro_rules! methods {
    ($($m:item,)*) => { () }
}

#[derive(Foo)]
struct S {
    f: Bar,
    methods! {
        fn baz() {}
    }
}

Where Foo expands to the struct and an impl wrapping the methods. This is clearly a hack, but I believe is used in the wild already.

jseyfried · 2018-03-28T18:27:13Z

@nrc

Ok, expand instead of opt-in makes sense. Do you think derives will ever be able to opt out of today's cfg expansion then?

This is clearly a hack, but I believe is used in the wild already.

Interesting, ideally I would have hoped "inert attributes" would be used for that:
#[derive(Foo)] #[methods(...)] struct S { ... } (c.f. serde).

nrc · 2018-03-29T15:52:02Z

Do you think derives will ever be able to opt out of today's cfg expansion then?

Hmm, this is an excellent question. I'm not really sure - I don't see why from a design perspective, however, from an implementation point of view messing with the order of 'expansion' of cfgs c.f. macros might be hard.

Interesting, ideally I would have hoped "inert attributes" would be used for that

iiuc, the problem there is that writing a lot of code inside an attribute is not so nice to look at.

jseyfried · 2018-03-29T17:51:36Z

from an implementation point of view messing with the order of 'expansion' of cfgs c.f. macros might be hard

Nah, it'd be pretty straightforward -- quite a bit simpler than this PR.

iiuc, the problem there is that writing a lot of code inside an attribute is not so nice to look at.

Yeah... I think with the right formatting it'd still be nicer, but if that's what's being done in the wild then so be it.

aidanhs · 2018-03-31T23:29:05Z

Hi @petrochenkov (crater requester/PR reviewer)! Crater results are at: http://cargobomb-reports.s3.amazonaws.com/pr-48465/index.html. 'Blacklisted' crates (spurious failures etc) can be found here. If you see any spurious failures not on the list, please make a PR against that file.

(interested observers: Crater is a tool for testing the impact of changes on the crates.io ecosystem. You can find out more at the repo if you're curious)

abonander · 2018-04-01T00:00:33Z

We've got some significant breakages in derive crates:

Several of these derives were expecting stringify!() in their input. They are probably expanding it internally anyways, so this change would probably be beneficial to them, but it looks like making this opt-in is our only option to avoid breakage.

petrochenkov · 2018-04-01T23:20:29Z

@abonander
So, should we close this PR then?
I think the opt-it/opt-out scheme deserves an RFC.
Also, the expansion order (as it works now) badly needs documentation - this is something that can be solved by the same RFC as well.

abonander · 2018-04-01T23:27:31Z

Thing with RFCs is that they're primarily for proposing solutions and I don't think we've gotten that far. I think exposing input expansion programmatically (in libproc_macro) is the most future-forward approach since it allows bang and attribute macros to utilize it as well. And it wouldn't take any effort to feature-gate it. Either way we should probably move this discussion to IRLO and close unless someone suggests something immediately actionable.

petrochenkov · 2018-04-01T23:35:28Z

Either way we should probably move this discussion to IRLO

Yeah, a pre-RFC on IRLO would probably be even better.

abonander · 2018-04-02T11:07:32Z

Created a topic on IRLO: https://internals.rust-lang.org/t/pre-rfc-input-expansion-for-proc-macros-derives-primarily/7210

rust-highfive assigned nikomatsakis Feb 23, 2018

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Feb 23, 2018

rust-highfive assigned nrc and unassigned nikomatsakis Feb 23, 2018

abonander changed the title ~~[WIP] expand items before their derives~~ Expand items before their derives Feb 24, 2018

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Feb 27, 2018

abonander force-pushed the expansion_order branch from 31c2fae to bf39cb5 Compare February 27, 2018 04:34

petrochenkov self-assigned this Feb 27, 2018

petrochenkov added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Feb 27, 2018

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Mar 3, 2018

petrochenkov added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Mar 3, 2018

abonander force-pushed the expansion_order branch from 228c8cb to f33551c Compare March 4, 2018 09:34

petrochenkov reviewed Mar 4, 2018

View reviewed changes

petrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-crater Status: Waiting on a crater run to be completed. labels Apr 1, 2018

petrochenkov closed this Apr 1, 2018

Expand items before their derives #48465

Expand items before their derives #48465

Conversation

abonander commented Feb 23, 2018 • edited Loading

rust-highfive commented Feb 23, 2018

abonander commented Feb 23, 2018 • edited Loading

abonander commented Feb 24, 2018

abonander commented Feb 25, 2018

bors commented Feb 27, 2018

abonander commented Feb 27, 2018

petrochenkov commented Mar 1, 2018

bors commented Mar 3, 2018

petrochenkov Mar 4, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

petrochenkov commented Mar 4, 2018 • edited Loading

abonander commented Mar 19, 2018 • edited Loading

aidanhs commented Mar 20, 2018

petrochenkov commented Mar 20, 2018

abonander commented Mar 20, 2018

petrochenkov commented Mar 20, 2018

keeperofdakeys commented Mar 20, 2018

abonander commented Mar 20, 2018

aidanhs commented Mar 26, 2018

nikomatsakis commented Mar 26, 2018

jseyfried commented Mar 26, 2018

jseyfried commented Mar 26, 2018

nrc commented Mar 27, 2018

abonander commented Mar 27, 2018

jseyfried commented Mar 27, 2018

abonander commented Mar 27, 2018

jseyfried commented Mar 27, 2018 • edited Loading

abonander commented Mar 27, 2018 • edited Loading

nrc commented Mar 28, 2018

jseyfried commented Mar 28, 2018

nrc commented Mar 29, 2018

jseyfried commented Mar 29, 2018

aidanhs commented Mar 31, 2018

abonander commented Apr 1, 2018 • edited Loading

petrochenkov commented Apr 1, 2018 • edited Loading

abonander commented Apr 1, 2018

petrochenkov commented Apr 1, 2018

abonander commented Apr 2, 2018

abonander commented Feb 23, 2018 •

edited

Loading

abonander commented Feb 23, 2018 •

edited

Loading

petrochenkov Mar 4, 2018 •

edited

Loading

petrochenkov commented Mar 4, 2018 •

edited

Loading

abonander commented Mar 19, 2018 •

edited

Loading

jseyfried commented Mar 27, 2018 •

edited

Loading

abonander commented Mar 27, 2018 •

edited

Loading

abonander commented Apr 1, 2018 •

edited

Loading

petrochenkov commented Apr 1, 2018 •

edited

Loading