async and other Rust 2018+ keywords not recognized as so in Rustdoc highlighting

[ThePuzzlemaker]

I tried this code:

/// ```rust ignore
/// fn foo(bar: &dyn Baz);
/// async fn quux() {
///     qaz.frobnify().await;
/// }
/// fn bar() {
///     let mut a = 0;
///     abstract blah; /// is a reserved keyword
/// }
/// ```
fn foo() {}

I expected to see this happen: async and other Rust 2018+ keywords are highlighted as my project is set to edition = "2018" in its Cargo.toml

Instead, this happened: They were not highlighted, but regular keywords (including reserved ones) were.

I believe that the reason why this is happening is this snippet in librustdoc:

rust/src/librustdoc/html/highlight.rs

Lines 292 to 305 in eb4fc71

	TokenKind::Ident => match text {
	"ref" | "mut" => Class::RefKeyWord,
	"self" | "Self" => Class::Self_,
	"false" | "true" => Class::Bool,
	"Option" | "Result" => Class::PreludeTy,
	"Some" | "None" | "Ok" | "Err" => Class::PreludeVal,
	// Keywords are also included in the identifier set.
	_ if Ident::from_str(text).is_reserved() => Class::KeyWord,
	_ if self.in_macro_nonterminal => {
	self.in_macro_nonterminal = false;
	Class::MacroNonTerminal
	}
	_ => Class::Ident,
	},

In this case, Ident::from_str will only create a dummy span (which isn't set to 2018 edition), so is_reserved does not include 2018+ keywords.

I would be willing to try to work on a PR to fix this issue if it is deemed worthy.

Meta

Tested

rustc --version --verbose:
stable:

rustc 1.48.0 (7eac88abb 2020-11-16)
binary: rustc
commit-hash: 7eac88abb2e57e752f3302f02be5f3ce3d7adfb4
commit-date: 2020-11-16
host: x86_64-unknown-linux-gnu
release: 1.48.0
LLVM version: 11.0

nightly:

rustc 1.50.0-nightly (7efc097c4 2020-12-12)
binary: rustc
commit-hash: 7efc097c4fe6e97f54a44cee91c56189e9ddb41c
commit-date: 2020-12-12
host: x86_64-unknown-linux-gnu
release: 1.50.0-nightly

Backtrace is not applicable.

[ThePuzzlemaker]

@rustbot label +T-rustdoc +A-rustdoc-ui +A-async-await
Hope these labels are accurate and helpful.

[tmandry]

@ThePuzzlemaker I think you have the right idea, though someone from the Rustdoc team might also be able to help with the details (cc @jyn514).

A PR would be welcome! Feel free to claim the issue with @rustbot claim. If you have questions, I'm also available to help here or on Zulip or DIscord, but my availability may be spotty over the next week.

[ThePuzzlemaker]

@rustbot claim

[jyn514]

Yup, that looks right, good catch! The only tricky thing will be passing in a span in to Classifier::new, but I think all the callsites have access to at least a Session, and you can get an edition from there.

[ThePuzzlemaker]

@rustbot release-assignment
I'm going to release assignment as I'm still not quite sure how this could be achieved even after looking through a bit of code.

[jyn514]

@ThePuzzlemaker where did you get stuck? I might be able to help.

[ThePuzzlemaker]

@jyn514 I couldn't find a way to get access to a Session or how I even get a Span from the lexer. I guess I'll reclaim though.
@rustbot claim

[jyn514]

@ThePuzzlemaker you need to follow the call chain up until you get to somewhere that has a Session. If nothing else, run_renderer has one, but I bet you can get it a lot more easily if you find a function with a context of some kind; then just pass the session as an argument until it gets to where it needs to be. (Alternatively, you could just pass the edition instead.)

[ThePuzzlemaker]

I guess if I just add it to the Session::new function then I can just follow the errors up.

[jyn514]

Wait, I'm confused - Session::new isn't part of rustdoc, and the code constructing a session will not be in rustdoc code. I recommend using rust-analyzer and go-to-definition to find all the callers of a function.

[ThePuzzlemaker]

I mean Classifier::new or whatever

[ThePuzzlemaker]

It looks like you can change the edition of a code block so I'll just do something like this:

• Highlight 2018 keywords if edition is explicitly set to 2018, don't if it's explicitly set to not 2018
• Highlight 2018 keywords if edition is set to 2018 for a crate and it's not explicitly set in the code block

[ThePuzzlemaker]

Hm, looks like I'm stuck again. I was able to pass an Edition to the Classifier but I don't know how to check Ident.is_reserved with an arbitrary edition (as to change the edition you'd have to make a Span, but I don't know how to get one for this case, and using a dummy span probably wouldn't work as you'd need a SyntaxContext to make a dummy span with a specific edition and I don't know how to get that either).

[ThePuzzlemaker]

I guess I could make a function Ident.is_reserved_for_edition and then have Ident.is_reserved just call that with the edition of the span in the ident (as since edition 2021 is around the corner, might as well make it a bit forwards-compatible 🙂)

[ThePuzzlemaker]

:) my changes worked, now to test and make sure they didn't bork anything else
edition = "2015":

edition = "2018":