Highlight edition-specific keywords correctly in code blocks, accounting for code block edition modifiers #80226

ThePuzzlemaker · 2020-12-20T08:35:28Z

Previously, edition-specific keywords (such as async and await) were not highlighted in code blocks, regardless of what edition was set. With this PR, this issue is fixed.

Now, the following behavior happens:

When a code block is explicitly set to edition X, keywords from edition X are highlighted
When a code block is explicitly set to a version that does not contain those keywords from edition X (e.g. edition Y), keywords from edition X are not highlighted
When a code block has no explicit edition, keywords from the edition passed via --edition to rustdoc are highlighted

For example, a project set with edition = "2015" in its Cargo.toml would not highlight async/await unless the code block was set to edition2018. Additionally, a project set with edition = "2018" in its Cargo.toml would highlight async/await unless the code block was set to a version that did not contain those keywords (e.g. edition2015).

This PR fixes #80004.

r? @jyn514

rust-log-analyzer · 2020-12-20T09:07:20Z

The job x86_64-gnu-llvm-9 failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

.................................................................................................... 9000/11189
.................................................................................................... 9100/11189
................................................................................i......i............ 9200/11189
.................................................................................................... 9300/11189
...................iiiiii..iiiiii.i................................................................. 9400/11189
.................................................................................................... 9600/11189
.................................................................................................... 9700/11189
..........................................................................................test [ui] ui/issues/issue-74564-if-expr-stack-overflow.rs has been running for over 60 seconds
.......... 9800/11189
---
Suite("src/test/assembly") not skipped for "bootstrap::test::Assembly" -- not in ["src/tools/tidy"]
Check compiletest suite=assembly mode=assembly (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)

running 27 tests
iiiiiiiiiiiiiiiiiiiiiiiiiii

Suite("src/test/incremental") not skipped for "bootstrap::test::Incremental" -- not in ["src/tools/tidy"]
 finished in 0.071 seconds
Check compiletest suite=incremental mode=incremental (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
---
Suite("src/test/debuginfo") not skipped for "bootstrap::test::Debuginfo" -- not in ["src/tools/tidy"]
Check compiletest suite=debuginfo mode=debuginfo (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)

running 116 tests
iiiiiiiiii.i.i...i..i..ii....ii....ii..........iiii.........i.....i...i.......ii.i.ii.....iiii.....i 100/116
test result: ok. 78 passed; 0 failed; 38 ignored; 0 measured; 0 filtered out; finished in 2.39s

 finished in 2.458 seconds
Suite("src/test/ui-fulldeps") not skipped for "bootstrap::test::UiFullDeps" -- not in ["src/tools/tidy"]
---
   Compiling once_cell v1.4.1
   Compiling expect-test v1.0.1
   Compiling rustdoc v0.0.0 (/checkout/src/librustdoc)
error[E0061]: this function takes 3 arguments but 2 arguments were supplied
  --> src/librustdoc/html/highlight/tests.rs:21:9
   |
21 |         write_code(&mut out, src);
   |         |
   |         expected 3 arguments
   |
note: function defined here
note: function defined here
  --> src/librustdoc/html/highlight.rs:50:4
   |
50 | fn write_code(out: &mut String, src: &str, edition: Edition) {

error[E0061]: this function takes 3 arguments but 2 arguments were supplied
error[E0061]: this function takes 3 arguments but 2 arguments were supplied
  --> src/librustdoc/html/highlight/tests.rs:33:5
33 |     write_code(&mut html, src);
   |     ^^^^^^^^^^ ---------  --- supplied 2 arguments
   |     |
   |     expected 3 arguments
   |     expected 3 arguments
   |
note: function defined here
  --> src/librustdoc/html/highlight.rs:50:4
   |
50 | fn write_code(out: &mut String, src: &str, edition: Edition) {

error: aborting due to 2 previous errors

For more information about this error, try `rustc --explain E0061`.
For more information about this error, try `rustc --explain E0061`.
error: could not compile `rustdoc`

To learn more, run the command again with --verbose.


command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "test" "--target" "x86_64-unknown-linux-gnu" "-Zbinary-dep-depinfo" "-j" "16" "--release" "--locked" "--color" "always" "--manifest-path" "/checkout/src/tools/rustdoc/Cargo.toml" "-p" "rustdoc:0.0.0" "--" "--quiet"


failed to run: /checkout/obj/build/bootstrap/debug/bootstrap --stage 2 test --exclude src/tools/tidy
Build completed unsuccessfully in 0:29:20

jyn514 · 2020-12-20T14:27:37Z

Hmm, I see - the issue is that the edition is not necessarily related to the span, so even if you pass in item.source, someone could have an edition2018 lang string on the code block.

The rustdoc side of things seems reasonable to me but I think this should have a reviewer from T-compiler.

r? @petrochenkov

jyn514 · 2020-12-20T14:28:25Z

Also you need to fix the unit tests:

error[E0061]: this function takes 3 arguments but 2 arguments were supplied
  --> src/librustdoc/html/highlight/tests.rs:21:9
   |
21 |         write_code(&mut out, src);
   |         ^^^^^^^^^^ --------  --- supplied 2 arguments
   |         |
   |         expected 3 arguments

ThePuzzlemaker · 2020-12-20T17:25:34Z

Hmm, I see - the issue is that the edition is not necessarily related to the span, so even if you pass in item.source, someone could have an edition2018 lang string on the code block.

@jyn514 I'm not sure what you're talking about. The only time I think I can see something like item.source is in this code: https://github.com/rust-lang/rust/blob/ffa98d6fb1144ca42ee92d0465b7bc1b669d64c1/src/librustdoc/html/render/mod.rs#L4717-L4728
I'm pretty sure that this code is only for highlighting macro definitions (e.g. at the top of dbg!).

ThePuzzlemaker · 2020-12-20T17:26:31Z

As for code blocks, they use either the explicit edition attribute or the edition given by --edition.
At least, that's how I've interpreted the code (please correct me though if I'm wrong).

compiler/rustc_span/src/symbol.rs

petrochenkov · 2020-12-21T17:59:38Z

One more question - I assumed that rustdoc "compiles" separate code blocks as independent programs, creating independent compilation sessions, so each session can have its own default edition possibly depending on the block's annotations (which will be used by Ident::from_str(text).is_reserved()).
Why doesn't it happen here?

ThePuzzlemaker · 2020-12-21T19:28:34Z

One more question - I assumed that rustdoc "compiles" separate code blocks as independent programs, creating independent compilation sessions, so each session can have its own default edition possibly depending on the block's annotations (which will be used by Ident::from_str(text).is_reserved()).
Why doesn't it happen here?

@petrochenkov I’m pretty sure this is just simple lexing for highlighting and rendering—individual sessions for each code block/doctest would probably be part of the doctest logic rather than rendering as that’d seem expensive for simple lexing.

rustc_span: Provide a reserved identifier check for a specific edition while keeping edition evaluation lazy because it may be expensive. Needed for rust-lang#80226.

ThePuzzlemaker · 2020-12-23T03:36:50Z

#80272 merged; no longer blocked. Will update and re-test.
@rustbot modify labels: -S-blocked

ThePuzzlemaker · 2020-12-23T04:45:15Z

Updated, rebased, and squashed. Working again 😄

ThePuzzlemaker · 2020-12-23T04:46:43Z

I used Symbol::intern(...).is_reserved rather than Ident::from_str(...).name.is_reserved to prevent unnecessary uses of a bunch of dummy spans

petrochenkov · 2020-12-23T08:19:04Z

@bors r=jyn514,petrochenkov

bors · 2020-12-23T08:19:05Z

📌 Commit d14ef17 has been approved by jyn514,petrochenkov

…yn514,petrochenkov Highlight edition-specific keywords correctly in code blocks, accounting for code block edition modifiers Previously, edition-specific keywords (such as `async` and `await`) were not highlighted in code blocks, regardless of what edition was set. With this PR, this issue is fixed. Now, the following behavior happens: - When a code block is explicitly set to edition X, keywords from edition X are highlighted - When a code block is explicitly set to a version that does not contain those keywords from edition X (e.g. edition Y), keywords from edition X are **not** highlighted - When a code block has no explicit edition, keywords from the edition passed via `--edition` to rustdoc are highlighted For example, a project set with `edition = "2015"` in its `Cargo.toml` would not highlight `async`/`await` unless the code block was set to `edition2018`. Additionally, a project set with `edition = "2018"` in its `Cargo.toml` *would* highlight `async`/`await` unless the code block was set to a version that did not contain those keywords (e.g. `edition2015`). This PR fixes rust-lang#80004. r? `@jyn514`

…yn514,petrochenkov Highlight edition-specific keywords correctly in code blocks, accounting for code block edition modifiers Previously, edition-specific keywords (such as `async` and `await`) were not highlighted in code blocks, regardless of what edition was set. With this PR, this issue is fixed. Now, the following behavior happens: - When a code block is explicitly set to edition X, keywords from edition X are highlighted - When a code block is explicitly set to a version that does not contain those keywords from edition X (e.g. edition Y), keywords from edition X are **not** highlighted - When a code block has no explicit edition, keywords from the edition passed via `--edition` to rustdoc are highlighted For example, a project set with `edition = "2015"` in its `Cargo.toml` would not highlight `async`/`await` unless the code block was set to `edition2018`. Additionally, a project set with `edition = "2018"` in its `Cargo.toml` *would* highlight `async`/`await` unless the code block was set to a version that did not contain those keywords (e.g. `edition2015`). This PR fixes rust-lang#80004. r? ```@jyn514```

Dylan-DPC-zz · 2020-12-24T18:16:49Z

has conflicts

@bors r-

ThePuzzlemaker · 2020-12-24T18:26:16Z

Rebased and fixed merge conflicts.

…, accounting for code block edition modifiers This is a squash of these commits: - Highlight edition-specific keywords correctly in code blocks, accounting for code block edition modifiers - Fix unit tests - Revert changes to rustc_span::symbol to prepare for merge of #80272 - Use new Symbol::is_reserved API from #80272 - Remove unused import added by accident when merging

ThePuzzlemaker · 2020-12-24T18:46:46Z

Fixed.

petrochenkov · 2020-12-24T19:09:56Z

@bors r=jyn514,petrochenkov

bors · 2020-12-24T19:09:57Z

📌 Commit db1451c has been approved by jyn514,petrochenkov

bors · 2020-12-25T11:16:58Z

⌛ Testing commit db1451c with merge ab10778...

bors · 2020-12-25T14:09:03Z

☀️ Test successful - checks-actions
Approved by: jyn514,petrochenkov
Pushing ab10778 to master...

rust-highfive assigned jyn514 Dec 20, 2020

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Dec 20, 2020

petrochenkov self-assigned this Dec 20, 2020

jyn514 added A-frontend Area: frontend (errors, parsing and HIR) T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Dec 20, 2020

rust-highfive unassigned jyn514 Dec 20, 2020

jyn514 added A-doctests Area: Documentation tests, run by rustdoc T-rustdoc Relevant to the rustdoc team, which will review and decide on the PR/issue. labels Dec 20, 2020

petrochenkov reviewed Dec 21, 2020

View reviewed changes

compiler/rustc_span/src/symbol.rs Outdated Show resolved Hide resolved

petrochenkov mentioned this pull request Dec 21, 2020

rustc_span: Provide a reserved identifier check for a specific edition #80272

Merged

petrochenkov added S-blocked Status: Marked as blocked ❌ on something else such as an RFC or other implementation work. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Dec 21, 2020

This comment has been minimized.

Sign in to view

rustbot removed the S-blocked Status: Marked as blocked ❌ on something else such as an RFC or other implementation work. label Dec 23, 2020

ThePuzzlemaker requested a review from petrochenkov December 23, 2020 06:05

bors added the S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. label Dec 23, 2020

GuillaumeGomez mentioned this pull request Dec 23, 2020

Rollup of 4 pull requests #80329

Closed

Dylan-DPC-zz mentioned this pull request Dec 23, 2020

Rollup of 14 pull requests #80343

Closed

Dylan-DPC-zz mentioned this pull request Dec 24, 2020

Rollup of 11 pull requests #80350

Closed

Dylan-DPC-zz mentioned this pull request Dec 24, 2020

Rollup of 11 pull requests #80355

Closed

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Dec 24, 2020

This comment has been minimized.

Sign in to view

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Dec 24, 2020

bors added the merged-by-bors This PR was explicitly merged by bors. label Dec 25, 2020

bors merged commit ab10778 into rust-lang:master Dec 25, 2020

rustbot added this to the 1.50.0 milestone Dec 25, 2020

ThePuzzlemaker deleted the issue-80004-fix branch December 25, 2020 14:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Highlight edition-specific keywords correctly in code blocks, accounting for code block edition modifiers #80226

Highlight edition-specific keywords correctly in code blocks, accounting for code block edition modifiers #80226

ThePuzzlemaker commented Dec 20, 2020

rust-log-analyzer commented Dec 20, 2020

jyn514 commented Dec 20, 2020

jyn514 commented Dec 20, 2020

ThePuzzlemaker commented Dec 20, 2020 •

edited

Loading

ThePuzzlemaker commented Dec 20, 2020 •

edited

Loading

petrochenkov commented Dec 21, 2020 •

edited

Loading

ThePuzzlemaker commented Dec 21, 2020

This comment has been minimized.

ThePuzzlemaker commented Dec 23, 2020 •

edited

Loading

ThePuzzlemaker commented Dec 23, 2020

ThePuzzlemaker commented Dec 23, 2020

petrochenkov commented Dec 23, 2020

bors commented Dec 23, 2020

Dylan-DPC-zz commented Dec 24, 2020

ThePuzzlemaker commented Dec 24, 2020

This comment has been minimized.

ThePuzzlemaker commented Dec 24, 2020

petrochenkov commented Dec 24, 2020

bors commented Dec 24, 2020

bors commented Dec 25, 2020

bors commented Dec 25, 2020

Highlight edition-specific keywords correctly in code blocks, accounting for code block edition modifiers #80226

Highlight edition-specific keywords correctly in code blocks, accounting for code block edition modifiers #80226

Conversation

ThePuzzlemaker commented Dec 20, 2020

rust-log-analyzer commented Dec 20, 2020

jyn514 commented Dec 20, 2020

jyn514 commented Dec 20, 2020

ThePuzzlemaker commented Dec 20, 2020 • edited Loading

ThePuzzlemaker commented Dec 20, 2020 • edited Loading

petrochenkov commented Dec 21, 2020 • edited Loading

ThePuzzlemaker commented Dec 21, 2020

This comment has been minimized.

ThePuzzlemaker commented Dec 23, 2020 • edited Loading

ThePuzzlemaker commented Dec 23, 2020

ThePuzzlemaker commented Dec 23, 2020

petrochenkov commented Dec 23, 2020

bors commented Dec 23, 2020

Dylan-DPC-zz commented Dec 24, 2020

ThePuzzlemaker commented Dec 24, 2020

This comment has been minimized.

ThePuzzlemaker commented Dec 24, 2020

petrochenkov commented Dec 24, 2020

bors commented Dec 24, 2020

bors commented Dec 25, 2020

bors commented Dec 25, 2020

ThePuzzlemaker commented Dec 20, 2020 •

edited

Loading

ThePuzzlemaker commented Dec 20, 2020 •

edited

Loading

petrochenkov commented Dec 21, 2020 •

edited

Loading

ThePuzzlemaker commented Dec 23, 2020 •

edited

Loading