resolve: Relax shadowing restrictions on macro-expanded macros #53778

petrochenkov · 2018-08-29T01:52:36Z

Previously any macro-expanded macros weren't allowed to shadow macros from outer scopes.
Now only "more macro-expanded" macros cannot shadow "less macro-expanded" macros.
See comments to fn may_appear_after and added tests for more details and examples.

The functional changes are a21f6f588fc28c97533130ae44a6957b579ab58c and 46dd365ce9ca0a6b8653849b80267763c542842a, other commits are refactorings.

rust-highfive · 2018-08-29T01:52:39Z

r? @eddyb

(rust_highfive has picked a reviewer for you, use r? to override)

petrochenkov · 2018-08-29T01:52:55Z

r? @alexcrichton or @oli-obk

petrochenkov · 2018-08-29T02:25:22Z

cc @nikomatsakis @nrc, you may be interested in this as well.
IIRC, there was a long discussion about what shadowing restrictions are necessary to make macro resolution/expansion predictable during the macro modularization RFC.

The resulting implemented rule was conservative - for macros 2.0 any macro-expanded macro could not shadow any other macro (legacy macros used more elaborate rules for backward compatibility).
This conservative rule prohibits totally reasonable code in which everything lives inside a single macro expansion (e.g. include!(...)) - see the example in #53205 (comment).

This PR refines that rule. The new rule is still simple enough (see the comments for fn may_appear_after) and is directly derived from requirements imposed on macro resolutions by any fixed-point expansion algorithm that's more or less similar to ours.
(It also pretty much fits what was previously implemented for legacy macros.)

alexcrichton · 2018-08-29T16:43:44Z

The code changes seems reasonable here, but I'm no expert on the intricacies of macro resolution nor what the long-term future plans are. In that sense I'll defer to @nrc and others for the review of the purpose of the PR

petrochenkov · 2018-08-29T17:01:42Z

r? @nrc @nikomatsakis

nrc · 2018-08-30T05:48:00Z

The code looks fine and the relaxed rules sound fine too, but I've really paged out name resolution, so I'm not 100% sure. Hopefully Niko can remember more than me.

petrochenkov · 2018-08-31T01:35:28Z

Updated with tests demonstrating possible expansion/shadowing configurations.
r? @nikomatsakis

resolve: Future proof resolutions for potentially built-in attributes Based on #53778 This is not full "pass all attributes through name resolution", but a more conservative solution. If built-in attribute is ambiguous with any other macro in scope, then an error is reported. TODO: Explain what complications arise with the full solution.

resolve: Future proof resolutions for potentially built-in attributes Based on #53778 This is not full "pass all attributes through name resolution", but a more conservative solution. If built-in attribute is ambiguous with any other macro in scope, then an error is reported. TODO: Explain what complications arise with the full solution. cc #50911 (comment) Closes #53531

nikomatsakis

I've not really read the code too deply, mostly the test cases. I'm not sure I really understand the rule yet! @petrochenkov maybe you can clarify?

nikomatsakis · 2018-09-04T16:30:15Z

src/librustc_resolve/macros.rs

+        //     define_mac!(); // if this generates another `macro_rules! mac`, then it can't shadow
+        //                    // the outer `mac` and we have and ambiguity error
+        //     mac!();
+        // }


Nice comment. One thing that is not clear to me — is this a change in semantics that is put in place by this PR? Or just documenting how things work today?

This particular snippet is documentation of existing behavior (it's copied from the function directly above).

Changes in semantics (mostly for modern macro macros) mostly appear when we put this snippet itself (or some parts of it) into a macro.

nikomatsakis · 2018-09-04T16:39:09Z

src/test/ui/macros/restricted-shadowing-legacy.rs

+// `+?` - configuration possible only with legacy scoping
+
+//  N | Outer ~ Invoc | Invoc ~ Inner | Outer ~ Inner | Possible |
+//  1 |       <       |       <       |       <       |    +     |


I don't really understand this chart. I would assume that this means that "Inner takes precedence" -- e.g., because Outer < Invoc. But check_1 below seems to report an error?

And what is N here?

Outer < Invoc means expansion producing Outer is less than expansion producing Invoc.
Expansions form a tree (code produced by a macro expansion may contains multiple nested expansions and so on), and < means strictly closer to the root of that tree.

N is just the number of combination, from 1 to 4 * 4 * 4 = 64.

Ah, ok, so this is like "can we construct a secnario where the outer is less than the invoc"... I see.

nikomatsakis · 2018-09-04T16:40:06Z

src/test/ui/macros/restricted-shadowing-legacy.rs

+
+        macro_rules! gen_inner_invoc { () => {
+            macro_rules! m { () => {} }
+            m!(); // OK


Presumably this expands to the m that is defined on the line before? is there some way we can test that...?

For example:

struct Right; struct Wrong; fn check5() -> Right { macro_rules! m { () => Wrong } macro_rules! gen_inner_invoc { () => { macro_rules! m { () => Right } m!() // OK }} gen_inner_invoc!() // Look ma, no type error... }

Hmm, yes, it's probably not too hard to insert such Wrongs for all "outer" shadowed macros, and Rights for "inner" macros, for all // OK cases.

nikomatsakis · 2018-09-04T16:41:56Z

src/test/ui/macros/restricted-shadowing-legacy.rs

+
+        macro_rules! gen_inner_gen_invoc { () => {
+            macro_rules! m { () => {} }
+            gen_invoc!(); // OK


Same resolution as before, I presume

nikomatsakis · 2018-09-04T16:45:34Z

src/librustc_resolve/lib.rs

+    // Suppose that we resolved macro invocation with `invoc_id` to binding `binding` at some
+    // expansion round `max(invoc_id, binding)` when they both emerged from macros.
+    // Then this function returns `true` if `self` may emerge from a macro *after* that
+    // in some later round and screw up our previously found resolution.


Hmm, can we improve this comment a bit?

For example, I am not clear on how you get an error -- the comment makes it sounds like returning true means that the result "may screw up our previously found resolution" -- so is that an error? Or does "may" here mean "is allowed to".

From inspecting the rest of the code, it seems like true means error. If so, may I suggest not_shadowed_by as a possible name? Alternatively, invert the sense: name it is_shadowed_by.

I'd also like some comments that explain the logic here. Here is my attempt to figure it out. =) I think some examples are helpful, too, so I'll try to supply some.

This function defines the shadowing rules across macro invocations. The idea here is that if a macro definition defines a macro and then invokes it, we should resolve to the macro defined locally. So, for example, the following macro resolves just fine:

macro gen_inner_invoc() { macro m() {} // B m!(); // OK -- resolves to B }

This is true even if that macro is defined (or invoked) in a context that already contains a macro m:

macro m() { } // macro def'n A, shadowed by B above maco gen_inner_invoc() { /* .. as above .. */ } gen_inner_invoc!();

The general rule is that a macro B will shadow some "earlier" macro A if:

B is defined in a "subinvocation" of the one that defined A (as above)

B is XX (see my questions below)

OK, I think "shadowed by"/"not shadowed by" is not the right way to think about it, that also leads to confusion here and in #53778 (comment).

Macro from inner scope always shadows a macro from outer scope:

// Fully expanded macro m() {} // Outer { macro m(); // Inner, always shadows outer }

the question is whether an instance of shadowing is "good" or "bad", which is determined entirely by their relative expansion order (earlier/later relations on expansions).
I hoped to convey that in the comments to may_appear_after, but probably not as successfully as I thought.

I'll try to reread this comment and #53778 (comment) later this evening and say something more specific.

Yes, I.. well I'm not sure I see but I may be starting to see. =) I sort of had the sense I was thinking about it wrong. I will try to revisit this with this paragraph in mind.

nikomatsakis · 2018-09-04T17:18:49Z

src/librustc_resolve/lib.rs

+        let certainly_before_other_or_simultaneously =
+            other_parent_expansion.is_descendant_of(self_parent_expansion);
+        let certainly_before_invoc_or_simultaneously =
+            invoc_parent_expansion.is_descendant_of(self_parent_expansion);


I'm not sure I understand this. I think this is saying that the self would legally be shadowed by any resolution that happens in "a child" macro.. but is that right? I'm probably missing some bit of context here. I guess I'm imagining that this would apply in a case like:

macro m { .. } macro foo { () => m!() } foo!() { }

Here, the invocation foo! is a "sub-expansion" of the one that defined m -- but this seems to suggest that there hence m the decision to use m cannot be shadowed. I guess that means that (e.g.) we have "definition 1" here?

macro m { .. } // definition 1 macro gen_m { () => macro m { ... } } gen_m!(); // generates definition 2, possibly later macro foo { () => m!() // resolves to definition 1 } foo!() { }

nikomatsakis · 2018-09-04T17:22:18Z

I see that we are time sensitive here. If needed, I think we could go ahead and land this PR, but I would really like it if we can start to write down in a clearer way (apart from the source, ideally) our "name resolution model".

petrochenkov · 2018-09-05T00:45:47Z

@nikomatsakis
So, I tried to lay out all the reasoning step by step from scratch and now I'm questioning all I know.

I'm still sure what this PR does is no worse than what the compiler was doing before it, and is also better in some sense.
However, I'm no longer sure why we are doing this at all, and what strict guarantees restricted shadowing gives us compared to simply verifying that the initial resolution of a macro (actually used during expansion) and its second validating resolution (performed after all expansion is done) are the same.

I need one more evening.

petrochenkov · 2018-09-05T00:57:31Z

First part of my notes:

Preface 1, The end result of macro expansion.

// Everything was expanded
macro m() {} // An Outer One, incorrect solution
{
	macro m() {} // An Outer One, incorrect solution
	{
		macro m() {} // An Outer One, incorrect solution
		{
			macro m() {} // The Inner One, correct solution
			
			m!(); // Or rather "trace" of it, since it was expanded too
		}
	}
}

Preface 2, Two stages.

Each macro call m!() is resolved twice - 1) initial resolution and 2) validating resolution.

Initial resolution

Happens when the crate is incomplete, some items are missing (not produced by expansions yet).
We find some solution and proceed with expanding it, this decision cannot be reverted later on, expansion does not backtrack.
The found solution may be incorrect (one of the Outer Ones), because the crate is incomplete.

Validating resolution

Happens when everything is expanded, the crate is complete and all items are in place.
Perfomed on a "trace" recorded during the initial resolution and containing all the data necessary for resolution - name and location of the original macro invocation.
Certainly founds the correct solution (the Inner One).

Looking at the initial resolution in more detail

So, we have just resolved some macro invocation to some macro definition (maybe even incorrect), what does that mean?
It means a lot actually!
It means that the macro invocation was already produced.
It also mean that the macro definition was already produced too.
It means that all parents of the invocation (expansion <= parent_expansion(invoc)) or the definition (expansion <= parent_expansion(initial_resolution)) were already expanded too.
It means that this is the best solution from those existing at this point in time (multiple solutions could be produced by the same expansion and appear simultaneously).
It means that if our initial resolution is incorrect (an Outer One), then the correct resolution (the Inner One) cannot be produced neither by ancestor(invoc) nor by ancestor(initial_resolution).
"Produced neither by ancestor(invoc) nor by ancestor(initial_resolution)" is exactly what fn may_appear_after returns.

...

bors · 2018-09-05T09:58:28Z

☔ The latest upstream changes (presumably #53410) made this pull request unmergeable. Please resolve the merge conflicts.

estebank · 2018-09-05T16:38:15Z

src/librustc_resolve/lib.rs

            })
-        } else if is_glob1 {
+        } else if b1.is_glob_import() {
            Some(format!("consider adding an explicit import of `{}` to disambiguate", name))
        } else {
            None
        };

        let mut err = struct_span_err!(self.session, span, E0659, "`{}` is ambiguous", name);


Can we also add a span_label pointing at span, maybe duplicating the text from the error?

nikomatsakis · 2018-09-05T17:12:15Z

@petrochenkov

I am a bit surprised to here that shadowing is not the right way to think about it. I think that this problem arises, perhaps, because we are not considering a "rich enough" view of the expanded source. Perhaps if we included the "trace" of the macros that were expanded in our result, we could define shadowing rules?

I basically expect that we ought to be able to show a fully expanded source and say that the error occurs because some macro invocation has two candidates and neither of them shadows one another. Then we can define the rule that lets a macro invocation shadow another as some specific invocation site.

It seems like the answer is:

At the invocation site X, a macro definition Inner shadows a macro definition Outer if:
- The definitions of both Inner and Outer are ancestors of the invocation site X
- The definition of Outer is a strict ancestor of the definition of Inner
  - this is overly simplified; I'm ignoring block scoping here, which has to be taken into account

I say that something is an "ancestor" if, in the tree of invocations, it is an ancestor -- in other words, if some text A was expanded to include B, then A is an ancestor of B. Also, B is an ancestor of itself. A "strict ancestor" is not.

Applying that to this test you wrote:

    fn check1() {
        macro m() {} // candidate A
        {
            #[rustc_transparent_macro]
            macro gen_gen_inner_invoc() {
                gen_inner!(); // generates candidate B
                m!(); //~ ERROR `m` is ambiguous
            }
            gen_gen_inner_invoc!();
        }
    }

I believe that this fully expands to:

macro m { } // candidate A
{
  macro gen_gen_inner_invoc() { /* definition not relevant */ }
  gen_gen_inner_invoc!() as {
    gen_inner!() as {
      macro m { .. } // candidate B
    }

    m!() // ERROR
  }
}

Here we have an error because candidate B is not an ancestor of the invocation.

In contrast, in check5:

    fn check5() {
        macro m() {}
        {
            #[rustc_transparent_macro]
            macro gen_inner_invoc() {
                macro m() {}
                m!(); // OK
            }
            gen_inner_invoc!();
        }
    }

we fully expand to

macro m { } // candidate A
{
  macro gen_gen_inner_invoc() { /* definition not relevant */ }
  gen_inner_invoc!() as {
    macro m { .. } // candidate B

    m!() // ERROR
  }
}

Here it works out because both A and B are ancestors of the invocation site, and A is a strict ancestor of B.

In this example, my rule doesn't work, because I didn't account for blocking scoping:

    fn check10() {
        macro m() {}
        {
            macro m() {}
            gen_invoc!(); // OK
        }
    }

but I think you can extend my "tree" notion to include not just macro invocations but also block scoping, and then it works fine here.

This example check13

    fn check13() {
        macro m() {}
        {
            gen_inner!();
            #[rustc_transparent_macro]
            macro gen_invoc() { m!() } //~ ERROR `m` is ambiguous
            gen_invoc!();
        }
    }

is basically the same as check1 from the point of view of my check. The gen_inner! call would create a macro candidate B that is visible to the invocation but not an ancestor of it, so shadowing results in an error.

(I realize now I didn't define how a macro becomes visible, but that is basically the same as any other amount of lexical scoping, except that we traverse through macro invocations.)

Does this description of the rule make sense to you @petrochenkov ?

petrochenkov · 2018-09-08T12:32:46Z

@nikomatsakis
Updated.

petrochenkov · 2018-09-09T20:14:00Z

@bors r=nikomatsakis p=1
I'm going to interpret the semi-r+ in #53778 (comment) as r+, otherwise we'll have too much code to backport after beta (which is in a few days).

bors · 2018-09-09T20:14:02Z

📌 Commit 2dce377 has been approved by nikomatsakis

bors · 2018-09-09T20:22:00Z

💡 This pull request was already approved, no need to approve it again.

There's another pull request that is currently being tested, blocking this pull request: rustc_resolve: inject uniform_paths canaries regardless of the feature-gate, on Rust 2018. #54011

bors · 2018-09-09T20:22:01Z

📌 Commit 2dce377 has been approved by nikomatsakis

bors · 2018-09-09T21:01:02Z

⌛ Testing commit 2dce377 with merge 2d4e34c...

resolve: Relax shadowing restrictions on macro-expanded macros Previously any macro-expanded macros weren't allowed to shadow macros from outer scopes. Now only "more macro-expanded" macros cannot shadow "less macro-expanded" macros. See comments to `fn may_appear_after` and added tests for more details and examples. The functional changes are a21f6f588fc28c97533130ae44a6957b579ab58c and 46dd365ce9ca0a6b8653849b80267763c542842a, other commits are refactorings.

bors · 2018-09-09T23:25:48Z

☀️ Test successful - status-appveyor, status-travis
Approved by: nikomatsakis
Pushing 2d4e34c to master...

rust-highfive assigned eddyb Aug 29, 2018

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Aug 29, 2018

rust-highfive assigned alexcrichton and unassigned eddyb Aug 29, 2018

rust-highfive assigned nrc and unassigned alexcrichton Aug 29, 2018

petrochenkov force-pushed the shadrelax2 branch from 3530727 to 19aad80 Compare August 31, 2018 01:31

rust-highfive assigned nikomatsakis and unassigned nrc Aug 31, 2018

petrochenkov force-pushed the shadrelax2 branch from 19aad80 to 5bd1974 Compare August 31, 2018 22:03

petrochenkov mentioned this pull request Sep 3, 2018

resolve: Future proof resolutions for potentially built-in attributes #53913

Merged

nikomatsakis reviewed Sep 4, 2018

View reviewed changes

petrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Sep 5, 2018

estebank reviewed Sep 5, 2018

View reviewed changes

petrochenkov mentioned this pull request Sep 6, 2018

resolve: Recover from indeterminate macro resolutions more agressively #53587

Closed

petrochenkov force-pushed the shadrelax2 branch from 996f44c to 2dce377 Compare September 8, 2018 12:32

petrochenkov added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Sep 8, 2018

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Sep 9, 2018

bors merged commit 2dce377 into rust-lang:master Sep 9, 2018

petrochenkov mentioned this pull request Sep 19, 2018

Ambiguity between custom derive attributes and proc macro attributes #52226

Closed

petrochenkov mentioned this pull request Oct 5, 2018

Add extern crate items to extern prelude #54658

Merged

petrochenkov mentioned this pull request Nov 17, 2018

Stabilize extern_crate_item_prelude #56032

Merged

petrochenkov mentioned this pull request Jan 2, 2019

Allow shadowing of procedural macro generated macros #57283

Closed

petrochenkov deleted the shadrelax2 branch June 5, 2019 16:15

petrochenkov mentioned this pull request Jun 24, 2019

Ambiguity errors in 2018 uniform import paths are not technically necessary #56414

Closed

ehuss mentioned this pull request Sep 9, 2019

Document name resolution. rust-lang/reference#568

Open

29 tasks

petrochenkov mentioned this pull request Mar 14, 2020

Identity macro 'launders' hygiene information #59011

Closed

petrochenkov mentioned this pull request Jul 21, 2020

Glob Time Travel #74556

Closed

petrochenkov mentioned this pull request Oct 30, 2020

use dep1::foo as dep1 is considered ambiguous #77586

Closed

smmalis37 mentioned this pull request Apr 15, 2021

ICE: #[rustc_variance] on trait: asked to compute variance for wrong kind of item #79699

Closed

petrochenkov mentioned this pull request Jul 15, 2024

Importing an enum variant can create a conflict with a new prelude type #127738

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

resolve: Relax shadowing restrictions on macro-expanded macros #53778

resolve: Relax shadowing restrictions on macro-expanded macros #53778

petrochenkov commented Aug 29, 2018 •

edited

Loading

rust-highfive commented Aug 29, 2018

petrochenkov commented Aug 29, 2018

petrochenkov commented Aug 29, 2018 •

edited

Loading

alexcrichton commented Aug 29, 2018

petrochenkov commented Aug 29, 2018

nrc commented Aug 30, 2018

petrochenkov commented Aug 31, 2018

nikomatsakis left a comment

nikomatsakis Sep 4, 2018

petrochenkov Sep 4, 2018

nikomatsakis Sep 4, 2018

petrochenkov Sep 4, 2018

nikomatsakis Sep 4, 2018

nikomatsakis Sep 4, 2018

petrochenkov Sep 4, 2018 •

edited

Loading

nikomatsakis Sep 4, 2018

nikomatsakis Sep 4, 2018

nikomatsakis Sep 4, 2018

petrochenkov Sep 4, 2018 •

edited

Loading

nikomatsakis Sep 4, 2018

nikomatsakis Sep 4, 2018

nikomatsakis commented Sep 4, 2018

petrochenkov commented Sep 5, 2018

petrochenkov commented Sep 5, 2018 •

edited

Loading

bors commented Sep 5, 2018

estebank Sep 5, 2018

nikomatsakis commented Sep 5, 2018

petrochenkov commented Sep 8, 2018

petrochenkov commented Sep 9, 2018 •

edited

Loading

bors commented Sep 9, 2018

bors commented Sep 9, 2018

bors commented Sep 9, 2018

bors commented Sep 9, 2018

bors commented Sep 9, 2018

resolve: Relax shadowing restrictions on macro-expanded macros #53778

resolve: Relax shadowing restrictions on macro-expanded macros #53778

Conversation

petrochenkov commented Aug 29, 2018 • edited Loading

rust-highfive commented Aug 29, 2018

petrochenkov commented Aug 29, 2018

petrochenkov commented Aug 29, 2018 • edited Loading

alexcrichton commented Aug 29, 2018

petrochenkov commented Aug 29, 2018

nrc commented Aug 30, 2018

petrochenkov commented Aug 31, 2018

nikomatsakis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

petrochenkov Sep 4, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

petrochenkov Sep 4, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nikomatsakis commented Sep 4, 2018

petrochenkov commented Sep 5, 2018

petrochenkov commented Sep 5, 2018 • edited Loading

Preface 1, The end result of macro expansion.

Preface 2, Two stages.

Initial resolution

Validating resolution

Looking at the initial resolution in more detail

...

bors commented Sep 5, 2018

Choose a reason for hiding this comment

nikomatsakis commented Sep 5, 2018

petrochenkov commented Sep 8, 2018

petrochenkov commented Sep 9, 2018 • edited Loading

bors commented Sep 9, 2018

bors commented Sep 9, 2018

bors commented Sep 9, 2018

bors commented Sep 9, 2018

bors commented Sep 9, 2018

petrochenkov commented Aug 29, 2018 •

edited

Loading

petrochenkov commented Aug 29, 2018 •

edited

Loading

petrochenkov Sep 4, 2018 •

edited

Loading

petrochenkov Sep 4, 2018 •

edited

Loading

petrochenkov commented Sep 5, 2018 •

edited

Loading

petrochenkov commented Sep 9, 2018 •

edited

Loading