Stabilize extended_key_value_attributes

[jyn514]

Closes #44732. Closes #78835. Closes #82768 (by making it irrelevant).

Stabilization report

Summary

This stabilizes using macro expansion in key-value attributes, like so:

#[doc = include_str!("my_doc.md")]
struct S;

Eager macro expansion only happens in a limited number of built-in attributes, with #[doc] being the primary example that will benefit from this. Notably, the attributes path, crate_type, and recursion_limit do not support this because they need their values before or during expansion. It is also not supported with proc-macros.

See Petrochenkov's excellent blog post on internals
for alternatives that were considered and rejected ("why accept no more and no less?")

This has been available on nightly since 1.50 with no major issues.

Notes

Accepted syntax

The parser accepts arbitrary Rust expressions in this position, but any expression other than a macro invocation will ultimately lead to an error because it is not expected by the built-in expression forms (e.g., #[doc]). Note that decorators and the like may be able to observe other expression forms.

Expansion ordering

Expansion of macro expressions in "inert" attributes occurs after decorators have executed, analogously to macro expressions appearing in the function body or other parts of decorator input.

There is currently no way for decorators to accept macros in key-value position if macro expansion must be performed before the decorator executes (if the macro can simply be copied into the output for later expansion, that can work).

Test cases

• https://github.com/rust-lang/rust/blob/master/src/test/ui/attributes/key-value-expansion-on-mac.rs
• https://github.com/rust-lang/rust/blob/master/src/test/rustdoc/external-doc.rs

The feature has also been dogfooded extensively in the compiler and
standard library:

• Cleanup LLVM debuginfo module docs #83329
• Remove unnecessary forward_inner_docs hack #83230
• Improve lang item generated docs #82641
• Use #[doc = include_str!()] in std #80534

Implementation history

• Initial proposal: ICE in macro: doc meta with expr on an item, string concat, stringify!(...) #55414 (comment)
• Experiment to see how much code it would break: [experiment] Expand macros in inert key-value attributes #67121
• Preliminary work to restrict expansion that would conflict with this
  feature: Expand NtExpr tokens only in key-value attributes #77271
• Initial implementation: Accept arbitrary expressions in key-value attributes at parse time #78837
• Fix for an ICE: resolve/expand: Improve attribute expansion on macro definitions and calls #80563

Unresolved Questions

#83366 (comment) listed some concerns, but they have been resolved as of this final report.

Additional Information

There are two workarounds that have a similar effect for #[doc]
attributes on nightly. One is to emulate this behavior by using a limited version of this feature that was stabilized for historical reasons:

macro_rules! forward_inner_docs {
    ($e:expr => $i:item) => {
        #[doc = $e]
        $i
    };
}

forward_inner_docs!(include_str!("lib.rs") => struct S {});

The other is to use doc(include):

 #![feature(external_doc)]
 #[doc(include = "lib.rs")]
 struct S {}

The first works, but is non-trivial for people to discover, and
difficult to read and maintain. The second is a strange special-case for
a particular use of the macro. This generalizes it to work for any use
case, not just including files.

I plan to remove doc(include) when this is stabilized
(#82539). The forward_inner_docs
workaround will still compile without warnings, but I expect it to be
used less once it's no longer necessary.

[rust-highfive]

r? @steveklabnik

(rust-highfive has picked a reviewer for you, use r? to override)

[jyn514]

r? @petrochenkov (forgot to put it in the PR description, sorry).

I added T-lang and T-compiler since I think they may need to both sign off on the FCP? Feel free to remove one or the other if necessary.

[pksunkara]

Stabilizing this would not make #82768 irrelevant because of people trying to provide backward compatibility.

Clap's MSRV is 1.46.0. We would like to keep it the same and use the doc feature flag to conditionally enable the doc = include_str! when building docs. That way, MSRV is bumped to 1.53.0 only when building the docs.

[jyn514]

Clap's MSRV is 1.46.0. We would like to keep it the same and use the doc feature flag to conditionally enable the doc = include_str! when building docs. That way, MSRV is bumped to 1.53.0 only when building the docs.

@pksunkara see the workaround in #82768 (comment).

[camelid]

Closes #44732.

I don't think this should close that issue since this PR does not stabilize, deprecate, or remove doc(include). #82539 should be the one to close that issue.

[petrochenkov]

One is to emulate this behavior by using eager
expansion

This is not eager expansion, this is a limited version of the same extended_key_value_attributes that exists for historical reasons.

[petrochenkov]

There are two questions to answer/acknowledge before stabilizing this:

• Whether to stabilize #[attr = ARBITRARY_EXPRESSION] or macro calls only #[attr = mac::call!(a b c)].

  If arbitrary expressions are stabilized, then things like

  #[attr = {
      let a = 10;
      println!("{}", a + 11);
  }]

  become legal as well, but only syntactically (under cfg(FALSE)), they will still be rejected semantically.

  Even if we only stabilize macros, right now the compiler has to know that mac::call!(a b c) is an expression macro to expand it, but it's not something fundamental, we may be able to expand macros in context independent way in the future.

• Relative order of expanding macros in #[attr = mac!()] and in other places.

  Right now mac!() is expanded last of all the macro invocations below

  #[macro_attr1]
  #[doc = mac!()]
  #[macro_attr2]
  #[derive(MacroDerive1, MacroDerive2)]
  struct S;
  
  bar!();

  and not between macro_attr1 and macro_attr2 as some people could expect.

  However, this falls under the general rule telling that the expansion order of macros applied to different targets is unspecified, i.e. macro_attr1 and macro_attr2 are applied to the same target struct S so they are ordered, but mac!() is an entirely separate invocation similarly to e.g. bar!().
  The problem is only that the expansion order is easily discoverable by macro_attr(1,2) and MacroDerive(1,2) here, so it becomes fixed in practice.

  Expanding mac!() in a way making its result visible to macro_attr2 should be somewhat annoying from the implementation point of view (I didn't investigate details).

[camelid]

• Expanding mac!() in a way making its result visible to macro_attr2 should be somewhat annoying from the implementation point of view (I didn't investigate details).

Do you mean macro_attr1 or do attribute macros evaluate from the outermost to the innermost? I would have assumed macro_attr1 would be applied on the result of doc; doc on the result of macro_attr; and macro_attr2 on the result of derive.

[petrochenkov]

@camelid
Macros on the same target are expanded in the left-to-right order, so #[macro_attr1] is run first and cannot see results of anything after it.

[webern]

I came here from the release notes for 1.54. Does this feature work for derive macros? It doesn't seem like it does. I'm trying something that looks like this:

#[some_derive_macro(some_attribute = myfoo!())]
struct Whatever {
    
}

Which does not work with Unable to parse attribute: expected literal.

What I actually want is this: #52393

And would look like this:

const MY_FOO: &str = "bloop";

#[some_derive_macro(some_attribute = MY_FOO)]
struct Whatever {
    
}

I thought I might be able to do a workaround using the new 1.54 feature:

macro_rules identity! { ($x:expr) => { $x }; }

const MY_FOO: &str = "bloop";

#[some_derive_macro(some_attribute = identity!(MY_FOO))]
struct Whatever {
    
}

Alas, I was naive.

Is there any chance that derive macros can get the same love as proc macros so that we can use macros (well, actually constants) as values in derive macro attribute values?

[jplatte]

This is not specifically about proc-macros: #78835 (comment)

[webern]

This is not specifically about proc-macros: #78835 (comment)

So if I read that correctly, this is supported:

#[some_derive_macro = something!()]
struct X {}

Bot not this:

#[some_derive_macro(attribute1 = something!())]
struct X {}

Is that what #78835 (comment) is saying?

Thanks.

[jyn514]

@webern Yes.

[michalfita]

@jyn514 but that doesn't:

#[path = concat!(env!("OUT_DIR"), "/lib.rs")]