Raw Keywords

[scottmcm]

Reserve k#keyword in edition 2021 and beyond as a general syntax for adding keywords mid-edition instead of needing speculative reservations.

Rendered

[scottmcm]

@danielhenrymantilla I think the answer there is that it fails to tokenize, like happens with m!( r#$pineapple );.

[danielhenrymantilla]

Nice 🙂 could you still mention that lexing requirement somewhere in the RFC?

[nikomatsakis]

@rfcbot fcp merge

Per the @rust-lang/lang team meeting today, this has been discussed for some time on Zulip and we have general consensus that we like this feature and would like to see it go through (though there are a few points of active discussion). This is somewhat time sensitive because we are trying to finalize the list of "edition candidate changes" and this would be on the list. Therefore, I'm going to fcp merge.

[rfcbot]

Team member @nikomatsakis has proposed to merge this. The next step is review by the rest of the tagged team members:

• @cramertj
• @joshtriplett
• @nikomatsakis
• @pnkfelix
• @scottmcm

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

[nikomatsakis]

For my part:

I think this helps address a crucial "tension" around editions -- I would like editions to come every few years, because I think more than that and it will be overwhelming, but I also want us to keep the train model working and having new features land.

I've also toyed with ideas like the ones that Ember has proposed, where you expose (and stabilize) "core building blocks" that let people build features in user-space before you stabilize the final user-facing form of the feature. I think that's awesome too, but I've found that for many of the features and RFCs I've looked at, its not obvious how to apply it. The k#foo seems to give us a useful tool for exposing and stabilizing features pre-edition.

(Thinking more about it, I realize these are two distinct things: the goal of the Ember form is to open up the design space for exploration by end-users, precisely for those cases where the final shape is not known; that's not really addressed by this RFC per se.)

[rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[joshtriplett]

I would very much like to see the 2015/2018 edition question settled in the RFC itself, rather than deferred to a "we could". I strongly suspect that "we could" will end up becoming "we didn't". I'd like the RFC to actually commit to addressing this, rather than deferring it to future work.

[joshtriplett]

@scottmcm Thank you, I really appreciate it. And I agree entirely with the principle you stated: "It's better for raw keywords to be nice on 2021 than for them to be consistent with 2015".

[kennytm]

i'm firmly on -1 side if you're going to introduce both r#$keyword and k#keyword in 2021.

the purpose of this RFC is to "reserve a (non-contextual) keyword" until the next edition comes, so why do we care so much about the aesthetics of a temporary syntax? it is especially silly to introduce two different syntaxes in pre-2018 (r#$) and post-2021 (k#), and even worse, making them simultaneously available albeit with lint.

imo you either

• (a) change the meaning of k#something in pre-2018 too (after crater), just like what you've already done with $(,)?+ (Extending ? macro repetition to Rust 2015 rust#56668),

• (b) accept that a raw keyword uses three tts (may break quote). in fact i've yet to see any rationale against multi-tt "keywords" such as do catch beside "this is ugly". you don't have the equivalent of $x:ident for keywords forcing a single-tt parse unlike identifiers.

  T-lang: can we get raw keywords
  me: we have raw keywords at home
  raw keywords at home:

  do async {
     do trust_me {
         read().do await?.copy_into(&do move sink).do await?;
     }
  } 

• (c) just use r#$ for post-2021 (-0 for this)

• (d) consider other alternative syntax not involving # such as k`something`

[nikomatsakis]

Hmm. @kennytm I'm not sure if this is exactly what you were proposing, but I have to admit that using do foo as the "way to write the keyword foo in any edition" has some appeal. It's a bit weird but...

do async fn foo() {
}

..reads nicely compared to...

k#async fn foo() {
}

The major problem I see there is that it's not a single token. I suspect this will have implications. It'd be a pain to integrate into the parser.

[scottmcm]

FWIW, I think the motivation and rationale here is strong, but I'm not at all tied to the specific form proposed. It's picked basically as the simplest thing that's evocative of raw idents, but I'd be happy to switch to a better lexical or syntactic form if one is found. (And this could include things like a new keyword in the edition as the normal way to do this.)

I'm not a big fan of foo.do await.bar, though, which makes me skeptical of anything with a space. There are a bunch of easy fixes for that, though, like foo.do-await.bar or foo.do.await.bar or do@yeet e or ...

[kennytm]

foo.do-await.bar

FYI Java does have introduced hyphenated keywords, e.g.

public non-sealed class WeirdShape extends Shape { ... }
//     ^~~~~~~~~~

[kennytm]

@nikomatsakis my preference, in order

1. if crater accepts k#something for 2015 and 2018, and we think that it is a valuable breakage (like that of $(,)?+), then make k#something available in 2015, 2018 too as a single token
2. if we don't want to break macros, make k # something a keyword. But I'm afraid this will break quote when people uses quote!(#k #something).
3. do something, do#something, do-something, etc.
4. r#$something
5. k`something` or `something` (wasting the ` syntax)

[nikomatsakis]

do-foo as a single token has some appeal. It would work across all editions, too.

[nikomatsakis]

Er, wait, it probably has the same macro problems.

[joshtriplett]

If crater tells us we can do k# in older editions without breakage, I'd absolutely be in favor of that. If that would be @scottmcm's preference as well, I'd be fine with the RFC stating that and only using r#$ as a previous-edition fallback if k# doesn't work.

I don't think anything along the lines of do would be worthwhile; it doesn't read well to me, and I find it harder to mentally lex. I'd prefer k#.

[bstrie]

I would like to propose one last addendum to this RFC: eliminate all currently-reserved-but-unused keywords.

Currently (stable) Rust reserves but does not use the following keywords:

abstract
become
box
do
final
macro
override
priv
typeof
unsized
virtual
yield

Since the mechanism introduced in this RFC is intended to obviate the need for such speculatively-reserved keywords, it seems reasonable to dogfood the feature by un-reserving these keywords.

(Of course, some of these keywords are in implemented-but-unstable features, e.g. macro, and others are in implemented-but-probably-never-going-to-be-stabilized features, e.g. box. The ones with reasonable paths to stability could potentially remain reserved.)

[bstrie]

(It occurs to me that one reason to support this feature on earlier editions is, ironically, macros: if we accept my prior comment then the definition of vec!, which uses box internally, would have to start using k#box, which means any code on earlier editions that uses the vec! macro would fail to compile if it did not understand the syntax. It also suggests that k#$foo is insufficient; the same syntax needs to be understood by all editions.)

[scottmcm]

@bstrie As unreserving can be done at any time (and has been, such as #2421), I'd prefer to keep that conversation out of a want-this-for-the-edition RFC.

[nikomatsakis]

I've given this some thought and I think I've come back to preferring k#foo over do-foo. I will say that in the end I could go either way, I don't think it matters all that much. I really want some way that allows us to add keywords in an "awkward but temporary" way which becomes nicer after the edition; whether it's k#foo or do-foo doesn't seem as important.

My reasoning, roughly in order of importance to me:

• Implementing do-foo seems a lot harder and it still has compatibility concerns. We have to contend with three distinct tokens that will become one.
• k#foo (to me eyes) signals pretty clearly "something wacky is going on" and will prompt folks to learn about what it is. They will be happy when the new edition comes because they can write a regular keyword. (I expect people heavily using k# keywords will actually wind up using procedural macros instead, which is also fine.)
• I agree with @scottmcm that spaces like do await don't seem right; we considered foo await and ultimately opted against it in favor of foo.await because it led to confusing "grouping" and I think the same concerns apply. do-await seems like it could have the same sort of confusion for readers who interpret - as a minus sign.
• k#foo is analogous to r#foo and fits with a general push to reserve [a-zA-Z_]+#identifier
• I don't really have strong opinions about editions prior to Rust 2021; I suspect we can use k#foo there in practice, and if not, we can use r#$foo or nothing, and I can live any of those choices. I don't find the counterarguments to r#$foo especially persuasive. It's a corner of the language, it won't affect users day to day, and it doesn't bring a lot of implementation complexity.

[bstrie]

I've filed RFC 3101 for syntactically reserving foo#bar for future use by the language starting with the 2021 edition: #3101 . If accepted, it would remove the time sensitivity of this RFC. However, RFC 3101 does nothing for pre-2021 editions, so that aspect of this RFC would remain to be resolved (however, there would seemingly no longer be any push to reach a decision here urgently).

[withoutboats]

I don't think this feature sounds very useful for its stated purpose.

I wouldn't describe this syntax as "slightly less nice" than actually having these reserved as keywords; I would describe it as substantially worse. If I were driving a feature that requires new keywords (like async/await), I would be strongly against stabilizing that feature before the most recent edition has those keywords as fully reserved. Or I would be if this feature were the only alternative; I would be searching for some alternative syntax we could use instead of keywords if I really wanted to ship before that edition. So if async/await had not been reserved in 2018, I would have found it unacceptably user-unfriendly to expect everyone to write k#async fn and .k#await when the feature stabilized.

I don't think the 3 year cadence is actually very problematic. async/await was the major feature we shipped the fastest. Let's say I had picked it up a year later, in December 2018, instead of December 2017 (so right after the 2018 edition shipped). This is assuming that the feature's prehistory with Alex's version hadn't given us cause to reserve the keywords. Even then, it would not have been considered ready to ship until fall 2020 in this timeline. So in that hypothetical world, we'd be talking about delaying the feature less than a year; probably we'd be trying to push edition 2021 forward a few releases in the worst case to cut down the delay.

So to summarize: I have a hard time imagining that the Rust project would ever actually use this feature. I also think this has a real downside. Even if its never used, it'll end up as yet another weird token types to be enumerated in the back of some reference manual, the sort of thing that reminds me of the non-design I see in programming languages that evokes for me the manuals for my kitchen appliances.

This seems to be pushed really fast to meet the edition cut off for the tokenization changes. I'd encourage just doing #3101 for that purpose instead and letting this get more rounds of internal and external feedback instead of pushing this through in a week.

[Havvy]

With the syntax reserved, this doesn't actually need to be implemented until there's a keyword to be added, unless we want to ensure that previous editions can use async/await.

[programmerjake]

to avoid the issue with quote!(#k#var) changing how it's parsed, how about instead have the raw keyword syntax use the currently illegal token 'keyword' (or k'keyword'), where keyword is any identifier with more than one character? So, in the 2015 edition we could use await async like so:

pub 'async' fn my_fn() -> i32 {
    calculate().'await'
}

[JonSeverinsson]

I like the idea in general, but would want to bikeshed a bit about the syntax...

I would prefer k#keyword on all editions, if it passes crater, or else r#$keyword on all editions. Having different syntaxes on different editions completely defeats the secondary purpose of simplifying the mental model of the parser: With this in place, the parser only care about editions when deciding whether to parse an unprefixed identifier as a keywords instead of an identifier.

(Obviously there are other differences between editions, primarily in selecting prelude and determining lint levels (including making some lints a hard error on some editions but not other), but those shouldn't affect the parser.)

[ehuss]

Would you mind saying more why using #![feature] to enable new keywords (on nightly or stable) isn't possible or desirable? I believe that the parser in rustc currently does not know about features, but I do not see why it couldn't (except maybe complexity with macros).

[ehuss]

I feel like the word "slightly" here is a subjective assertion that does not match my view. Something like k#catch { ... } looks quite poor to me, and I think will make documentation and teaching more difficult. The k# prefix doesn't just go away during the next edition, it is around forever. I can imagine trying to explain these things to novice users could increase the load when trying to learn a language that is already quite syntax-heavy. It is also a small burden on tooling.

This also pre-supposes a 3-year cadence for editions, which is not mandated in #3085. My impression is that features that add new keywords usually take years to develop (in the design, RFC, implementation, testing, stabilization, documentation steps). I would think that if something of that magnitude is ready in say 2023, a new edition release could be coordinated to ship it that year to avoid making the feature wait until 2024.

[joshtriplett]

The k# prefix doesn't just go away during the next edition, it is around forever.

I'm not sure I follow that; for any given keyword, k# does go away during the next edition.

[scottmcm]

We discussed this RFC again in the lang team triage meeting today.

For the short-term goal of the reservation for the edition, we'll be moving forward on #3101 instead. As such, we wanted to leave more time for conversations about this one, and maybe use crater results from 3101 to make design changes,

@rfcbot cancel

[rfcbot]

@scottmcm proposal cancelled.

[nikomatsakis]

@ehuss

Regarding these two points...

Would you mind saying more why using #![feature] to enable new keywords (on nightly or stable) isn't possible or desirable? I believe that the parser in rustc currently does not know about features, but I do not see why it couldn't (except maybe complexity with macros).

It's not desirable, in my view, because it creates a situation where the set of keywords you can use in a program is not determined solely by the current edition, but also by the feature flags present on the crate. This means we wind up with a kind of 'combinatorics' of Rust dialects.

It's also true that the recognizing keywords is typically done at the lexer level, so there is a bit of a feedback issue between #![feature] declarations. Not unsolveable, but a pain.

These are the reasons that we opted to use edition = XXX in the cargo toml instead.

This also pre-supposes a 3-year cadence for editions, which is not mandated in #3085. My impression is that features that add new keywords usually take years to develop (in the design, RFC, implementation, testing, stabilization, documentation steps). I would think that if something of that magnitude is ready in say 2023, a new edition release could be coordinated to ship it that year to avoid making the feature wait until 2024.

The objective here, ultimately, is to disconnect the stabilization and progress of features from the edition schedule, in an effort to avoid developer stress. It's true that we could potentially have (e.g.) yearly edition, but that has non-trivial costs -- both in terms of public perception, but also in terms of developer stress brought on by having more regular deadlines. Putting together an edition is some amount of planning effort, even a relatively low-key one like Rust 2021.

[ehuss]

Thank you for responding. For me, it helps to explicitly write down reasons why alternatives are not chosen so that they don't remain as tribal knowledge. For example, I had a sense there is a discomfort with the way Haskell language extensions work, but I have not seen the reasons written down.

But perhaps I should have been clearer. From my perspective (as a user), there is already a combinatorics of syntax on nightly with the #![feature] attribute. Though box foo is technically allowed syntactically, in practice a user does not perceive this as valid syntax unless they are using feature(box_patterns). To me, there is a lot more to the syntax than just keywords and punctuation, so I don't separate keywords out as anything different from the other #![feature] features that change the language.

My intent was to say, allow new keywords to be enabled with #![feature] on nightly only, which relieves the pressure to try to reserve them during an edition release. Then, there is freedom to experiment, design, and implement new language features that require new keywords to be done at any time (just like other language changes do). When the next edition comes around, they can ride that train to stable if they are ready. Features that are ready long before the next edition release may have to wait a while, but I think it can be good to put some resistance on the pace that the language changes, especially for very large changes that require new keywords. I understand that the team prefers the worse syntax over waiting, but #3101 does not discuss this at all as an intentional decision.

I think some of the stress for editions is caused by the lack of treating it as a train. Although #3085 tries to frame it as a train release, in practice 2021 has not been that. It has been a rush to see what changes can be squeezed in, even if that means missing deadlines. Obviously that is due to the large difference in timeframes from regular Rust releases (and this being a volunteer process, ❤️s to everyone working on it). But I don't think it is fair to frame an edition as a train-style release vehicle when there is no prescribed schedule and things people really want are allowed to push deadlines. If the intent is to relieve stress, it may be worth examining the edition process more closely, as the rush will still happen for non-keyword changes.

[nikomatsakis]

For me, it helps to explicitly write down reasons why alternatives are not chosen so that they don't remain as tribal knowledge.

Yes! I really like to have a thorough explanation of the alternatives and the rationale around them. I thought that much of what I said was in alternatives section of the 2021 RFC, but I guess that's not entirely true.

My intent was to say, allow new keywords to be enabled with #![feature] on nightly only, which relieves the pressure to try to reserve them during an edition release.

This is already possible; I don't think we need an RFC for this. The point here is to permit stabilization, not nightly experimentation.

I guess the key question is precisely whether it's good to put back-pressure on potential new keywords that may be ready long before the edition release. I think we have plenty of checks that come before we get to the point of stabilization and I'm not overly concerned about things being stabilized willy-nilly.

To me, the strongest argument against this proposal is that it's incomplete; only some kinds of new features (those reliant on keywords) can be phased in this way. Things like RFC 2229 would not be applicable, unless we added an artificial keyword to use for opt-in. I'm not sure whether that's a problem or not, we'll have to see in time.

I do still think of Editions as trains, even if they are trains on an irregular schedule; I also agree that we've not been strict about the deadlines, but we've also tried to treat them meaningfully. We ruled out a number of things that didn't seem to be approaching consensus. We allowed the 'reserve syntactic space' to stay on the list because the team seemed to be in favor of the general idea and the remaining concerns seemed fairly narrow, almost like implementation details. However, we did agree in the last meeting that those really ought to be ironed out by the next meeting (minutes are here, though I don't think we've well publicized this).

[scottmcm]

With #3101 shipped today, and nothing really happening here, I'm going to close this for now -- we can always reopen later.

[joshtriplett]

This came up a couple of times, and several folks (myself, @nikomatsakis, and @pnkfelix) were all surprised to realize we'd never ratified it, let alone implemented it. This keeps coming up in discussions whenever we talk about needing a keyword for something, in order to make that feature available in all editions.

Nominating for discussion.

[workingjubilee]

I also found myself surprised when I recently reviewed a PR that stabilized something mostly only on an edition boundary.