Macro fragment specifiers edition policy

text/3531-macro-fragment-policy.md

@@ -0,0 +1,67 @@
+# Macro matcher fragment specifiers edition policy
+
+- Start Date: 2023-11-15
+- RFC PR: [rust-lang/rfcs#3531](https://github.com/rust-lang/rfcs/pull/3531)
+
+# Summary
+
+This RFC sets out the policy for how the behavior of macro matcher fragment specifiers is updated over an edition when those specifiers fall out of sync with the underlying grammar of Rust.
+
+# Background and motivation
+
+Rust has a syntactic abstraction feature called ["macros by example"][] or `macro_rules`.  This feature allows for writing *macros* that transform source code in a principled way.
+
+Each macro is composed of one or more *rules*.  Each of these rules has a *matcher*.  The matcher defines what pattern of Rust syntax will be matched by the rule.
+
+Within a matcher, different parts of the input Rust syntax can be bound to metavariables using *[fragment specifiers][]*.  These fragment specifiers define what Rust syntax will be matched and bound to each metavariable.  For example, the `item` fragment specifier matches an [item][], `block` matches a [block expression][], `expr` matches an [expression][], and so on.
+
+As we add new features to Rust, sometimes we change its syntax.  This means that, even within an edition, the definition of what exactly constitutes an [expression][], e.g., can change.  However, to avoid breaking macros in existing code covered by our stability guarantee, we do not update within an edition what code is matched by the relevant fragment specifier (e.g., `expr`).  This *skew* or divergence between the language and the fragment specifiers creates problems over time, including that macros become unable to match newer Rust syntax without dropping down to lower-level specifiers such as `tt`.
+
+Periodically, we need a way to bring the language and the fragment specifiers back into sync.  This RFC defines a policy for how we do that.
+
+["macros by example"]: https://doc.rust-lang.org/reference/macros-by-example.html
+[block expression]: https://doc.rust-lang.org/reference/expressions/block-expr.html
+[expression]: https://doc.rust-lang.org/reference/expressions.html
+[fragment specifiers]: https://doc.rust-lang.org/reference/macros-by-example.html#metavariables
+[item]: https://doc.rust-lang.org/reference/items.html
+
+# Policy
+
+This section is normative.
+
+When we change the syntax of Rust such that the syntax matched by a fragment specifier no longer exactly aligns with the actual syntax for that production in the Rust grammar, we will:
+
+- In all editions, add a new fragment specifier that preserves the behavior of the existing fragment specifier.  If there is some semantically meaningful name that makes sense to use for this new fragment specifier, we'll use that.  Otherwise, we'll use the existing name with the identifier of the current stable edition added as a suffix.
+- In the next edition, change the behavior of the original fragment specifier to match the underlying grammar as of the release of Rust corresponding to first release of that edition.
+- When migrating existing code to the new edition, have `cargo fix` replace all instances of the original fragment specifier in macro matchers with the new one that preserves the old behavior.
+
+For example, suppose that the current stable edition is Rust 2021, the behavior of the `expr` fragment specifier has fallen out of sync with the grammar for a Rust [expression][], and that Rust 2024 is the next edition.  Then in all editions of Rust, we would add a new fragment specifier named `expr2021` (assuming no better semantically meaningful name could be found) that would preserve the behavior `expr` had in Rust 2021, we would in Rust 2024 change the behavior of `expr` to match the underlying grammar, and when migrating code to Rust 2024, we would have `cargo fix` replace all instances of `expr` fragment specifiers with `expr2021`.
+
+A new fragment specifier that preserves the old behavior *must* be made available no later than the first release of Rust for the new edition, but it *should* be made available as soon as the original fragment specifier first diverges from the underlying grammar.
+
+# Alternatives
+
+## Keep the old, add specifiers for the new
+
+Changing the behavior of existing fragment specifiers, even over an edition, has an obvious cost: we may change the meaning of existing macros and consequently change the code that they generate.
+
+Having `cargo fix` replace all instances of a changed fragment specifier with the new fragment specifier added for backward compatibility does mitigate this.  But that has some cost in terms of code churn.
+
+Another alternative would be to *never* change the meaning of existing fragment specifiers.  Instead, when changing the grammar of Rust, we would add *new* fragment specifiers that would correspond with this new grammar.  We would not have to wait for new editions to add these.  We could add, e.g., `expr2023_11`, `expr2023_12`, etc. each time that we change the grammar.
+
+This would be burdensome in other ways, so we've decided not to do this.
+
+## Add specifier for new edition behavior in all editions
+
+In addition to doing what is specified in this RFC, when releasing a new edition we could also add a new fragment specifier to all editions whose behavior would match that of the original fragment specifier in the new edition.  E.g., when releasing Rust 2024, we would add an `expr2024` fragment specifier to all editions that would match the behavior of `expr` in Rust 2024.
+
+The upside of doing this would be that people could take advantage of the new behavior without migrating their crates to the new edition.  Conceivably, this could help to allow some crates to make incremental transitions.
+
+However, if later, during the life of the Rust 2024 edition, we were to change the grammar of expressions again and come up with a semantically meaningful name for the fragment specifier that would preserve the Rust 2024 behavior, then we would end up with two identical fragment specifiers for this, `expr2024` and `expr_some_better_name`.
+
+More importantly, making changed new edition behavior optionally available in older editions is not what we generally do.  As [RFC 3085][] said, [editions are meant to be adopted][].  The way for a crate to opt in to the behavior of the new edition is to upgrade to that edition.
+
+Consequently, we've decided not to do this.
+
+[RFC 3085]: https://github.com/rust-lang/rfcs/blob/master/text/3085-edition-2021.md
+[editions are meant to be adopted]: https://github.com/rust-lang/rfcs/blob/master/text/3085-edition-2021.md#editions-are-meant-to-be-adopted