Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't generate fake IDs in ReplaceBodyWithLoop #73532

Closed
wants to merge 4 commits into from

Conversation

jyn514
Copy link
Member

@jyn514 jyn514 commented Jun 20, 2020

Follow up to #73103 with less hackiness and special-casing.

Closes #71820, closes #71104. Should fix #73101, I'm testing locally now.

Currently this includes the changes from #73523 but I can remove those without trouble.

r? @ecstatic-morse

jyn514 and others added 4 commits June 19, 2020 20:06
It turns out that this has not been working for who knows how long.
Previously:

```
pub fn h() { 1 + 2; }
```

After this change:

```
pub fn h() { loop {} }
```

This only affected the pass when run with the command line
pretty-printing option, so rustdoc was still replacing bodies with
`loop {}`.
- x.py fmt
- Remove commented-out code
- Change variable names to match usage
- Fix indentation
@rust-highfive rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jun 20, 2020
@jyn514
Copy link
Member Author

jyn514 commented Jun 20, 2020

Most of visit_block was de-dented two levels so I recommend viewing with whitespace changes hidden.

@jyn514
Copy link
Member Author

jyn514 commented Jun 20, 2020

This did not fix the panics in #73101. I'm not sure why since it got rid of the fake parents. I think I might need to update the stmts to change the parent somehow, since nested items will be moved up several levels?

[DEBUG rustc_query_system::query::plumbing] ty::query::get_query<def_kind>(key=DefId(0:4 ~ macro_in_closure[8787]::main[0]::{{closure}}[0]), span=/home/joshua/rustc/src/test/rustdoc/macro-in-closure.rs:1:1: 1:1 (#0))
thread 'rustc' panicked at 'called `Option::unwrap()` on a `None` value', /home/joshua/src/rust/src/librustc_hir/definitions.rs:358:9

@jyn514
Copy link
Member Author

jyn514 commented Jun 20, 2020

For context, this is what everybody_loops does before the PR:

fn main() {
    {
        macro m { () => { } }
    }
    loop  { }
}

and this is what it does after:

fn main() {
    macro m { () => { } }
    loop  { }
}

@jyn514
Copy link
Member Author

jyn514 commented Jun 20, 2020

The other error is while documenting libcore:

 error[E0412]: cannot find type `BorrowedPlaceholder` in this scope
    --> src/libcore/fmt/mod.rs:2205:32
     |
2205 |                 impl Debug for BorrowedPlaceholder {
     |                                ^^^^^^^^^^^^^^^^^^^ not found in this scope

This is for the following snippet:

   2195 #[stable(feature = "rust1", since = "1.0.0")]
   2196 impl<T: ?Sized + Debug> Debug for RefCell<T> {
   2197     fn fmt(&self, f: &mut Formatter<'_>) -> Result {
   2198         match self.try_borrow() {
   2199             Ok(borrow) => f.debug_struct("RefCell").field("value", &borrow).f
   2199 inish(),
   2200             Err(_) => {
   2201                 // The RefCell is mutably borrowed so we can't look at its va
   2201 lue
   2202                 // here. Show a placeholder instead.
   2203                 struct BorrowedPlaceholder;
   2204 
   2205                 impl Debug for BorrowedPlaceholder {
   2206                     fn fmt(&self, f: &mut Formatter<'_>) -> Result {
   2207                         f.write_str("<borrowed>")
   2208                     }
   2209                 }
   2210 
   2211                 f.debug_struct("RefCell").field("value", &BorrowedPlaceholder   2211 ).finish()
   2212             }
   2213         }
   2214     }
   2215 }

I'm moving the impl up to the top level, but not the type. I need to bring the type up as well.

@rust-highfive
Copy link
Collaborator

The job x86_64-gnu-llvm-8 of your PR failed (pretty log, raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
##[section]Starting: Linux x86_64-gnu-llvm-8
##[section]Starting: Initialize job
Agent name: 'Azure Pipelines 5'
Agent machine name: 'fv-az619'
Current agent version: '2.170.1'
##[group]Operating System
16.04.6
LTS
LTS
##[endgroup]
##[group]Virtual Environment
Environment: ubuntu-16.04
Version: 20200614.1
Included Software: https://github.com/actions/virtual-environments/blob/ubuntu16/20200614.1/images/linux/Ubuntu1604-README.md
##[endgroup]
Agent running as: 'vsts'
Prepare build directory.
Set build variables.
Download all required tasks.
Download all required tasks.
Downloading task: Bash (3.163.3)
Checking job knob settings.
   Knob: AgentToolsDirectory = /opt/hostedtoolcache Source: ${AGENT_TOOLSDIRECTORY} 
   Knob: AgentPerflog = /home/vsts/perflog Source: ${VSTS_AGENT_PERFLOG} 
Start tracking orphan processes.
##[section]Finishing: Initialize job
##[section]Starting: Configure Job Name
==============================================================================
---
========================== Starting Command Output ===========================
[command]/bin/bash --noprofile --norc /home/vsts/work/_temp/e434bee3-6d38-453e-9128-4f97bf8fd733.sh

##[section]Finishing: Disable git automatic line ending conversion
##[section]Starting: Checkout rust-lang/rust@refs/pull/73532/merge to s
Task         : Get sources
Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
Version      : 1.0.0
Author       : Microsoft
---
##[command]git remote add origin https://github.com/rust-lang/rust
##[command]git config gc.auto 0
##[command]git config --get-all http.https://github.com/rust-lang/rust.extraheader
##[command]git config --get-all http.proxy
##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/73532/merge:refs/remotes/pull/73532/merge
---
 ---> 31fea614d2f3
Step 5/8 : ENV RUST_CONFIGURE_ARGS       --build=x86_64-unknown-linux-gnu       --llvm-root=/usr/lib/llvm-8       --enable-llvm-link-shared       --set rust.thin-lto-import-instr-limit=10
 ---> Using cache
 ---> 4195cadf126d
Step 6/8 : ENV SCRIPT python2.7 ../x.py test --exclude src/tools/tidy &&            python2.7 ../x.py test src/test/mir-opt --pass=build                                   --target=armv5te-unknown-linux-gnueabi &&            python2.7 ../x.py test src/tools/tidy
 ---> 4e90f6b48f05
Step 7/8 : ENV NO_DEBUG_ASSERTIONS=1
 ---> Using cache
 ---> dfa0a356d899
---
Set({"src/librustc_parse_format"}) not skipped for "bootstrap::test::CrateLibrustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_passes"}) not skipped for "bootstrap::test::CrateLibrustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_plugin_impl"}) not skipped for "bootstrap::test::CrateLibrustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_privacy"}) not skipped for "bootstrap::test::CrateLibrustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_query_system"}) not skipped for "bootstrap::test::CrateLibrustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_save_analysis"}) not skipped for "bootstrap::test::CrateLibrustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_serialize"}) not skipped for "bootstrap::test::CrateLibrustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_session"}) not skipped for "bootstrap::test::CrateLibrustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_span"}) not skipped for "bootstrap::test::CrateLibrustc" -- not in ["src/tools/tidy"]
---
Set({"src/librustc_parse_format"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_passes"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_plugin_impl"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_privacy"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_query_system"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_save_analysis"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_serialize"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_session"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_span"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
---
   Compiling rustc_parse_format v0.0.0 (/checkout/src/librustc_parse_format)
   Compiling chalk-rust-ir v0.10.0
   Compiling rustc_ast_pretty v0.0.0 (/checkout/src/librustc_ast_pretty)
   Compiling rustc_hir v0.0.0 (/checkout/src/librustc_hir)
   Compiling rustc_query_system v0.0.0 (/checkout/src/librustc_query_system)
   Compiling chalk-solve v0.10.0
   Compiling rustc_hir_pretty v0.0.0 (/checkout/src/librustc_hir_pretty)
   Compiling rustc_parse v0.0.0 (/checkout/src/librustc_parse)
   Compiling rustc_ast_lowering v0.0.0 (/checkout/src/librustc_ast_lowering)
---
   Compiling rustc_parse_format v0.0.0 (/checkout/src/librustc_parse_format)
   Compiling chalk-rust-ir v0.10.0
   Compiling rustc_ast_pretty v0.0.0 (/checkout/src/librustc_ast_pretty)
   Compiling rustc_hir v0.0.0 (/checkout/src/librustc_hir)
   Compiling rustc_query_system v0.0.0 (/checkout/src/librustc_query_system)
   Compiling chalk-solve v0.10.0
   Compiling rustc_hir_pretty v0.0.0 (/checkout/src/librustc_hir_pretty)
   Compiling rustc_parse v0.0.0 (/checkout/src/librustc_parse)
   Compiling rustc_ast_lowering v0.0.0 (/checkout/src/librustc_ast_lowering)
---
.................................................................................................... 1900/10338
.................................................................................................... 2000/10338
.................i..i............................................................................... 2100/10338
.................................................................................................... 2200/10338
.......iiiii........................................................................................ 2300/10338
.................................................................................................... 2500/10338
.................................................................................................... 2600/10338
.................................................................................................... 2700/10338
.................................................................................................... 2800/10338
---
.................................................................................................... 5300/10338
.........................................................................................i.......... 5400/10338
...................................................................................i................ 5500/10338
.................................................................................................... 5600/10338
..ii.ii........i...i................................................................................ 5700/10338
...............................................................i.................................... 5900/10338
.................................................................................................... 6000/10338
.................ii.....................................i........................................... 6100/10338
.................................................................................................... 6200/10338
.................................................................................................... 6200/10338
.................................................................................................... 6300/10338
................................................................................ii...i..ii.......... 6400/10338
.................................................................................................... 6600/10338
.................................................................................................... 6700/10338
.................................................................................................... 6800/10338
.................................................................................................... 6800/10338
..............i..ii................................................................................. 6900/10338
.................................................................................................... 7100/10338
......................................................................i............................. 7200/10338
.................................................................................................... 7300/10338
.................................................................................................... 7400/10338
---
.................................................................................................... 8200/10338
.................................................................................................... 8300/10338
.................................................................................................... 8400/10338
...............i.................................................................................... 8500/10338
.....................................................................iiiiii.iiiiii.i................ 8600/10338
..........................i......................................................................... 8800/10338
.................................................................................................... 8900/10338
.................................................................................................... 9000/10338
.................................................................................................... 9100/10338
---
Suite("src/test/codegen") not skipped for "bootstrap::test::Codegen" -- not in ["src/tools/tidy"]
Check compiletest suite=codegen mode=codegen (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)

running 196 tests
iiii......i...............ii.i..........i......................i...........i..i........i........i... 100/196
.i.............i.i.i...iii..iiii....................................iii.................ii......

 finished in 5.467
Suite("src/test/codegen-units") not skipped for "bootstrap::test::CodegenUnits" -- not in ["src/tools/tidy"]
Check compiletest suite=codegen-units mode=codegen-units (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
---
Suite("src/test/assembly") not skipped for "bootstrap::test::Assembly" -- not in ["src/tools/tidy"]
Check compiletest suite=assembly mode=assembly (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)

running 22 tests
iiiiiiiiiiiiiiiiiiiiii

 finished in 0.128
Suite("src/test/incremental") not skipped for "bootstrap::test::Incremental" -- not in ["src/tools/tidy"]
Check compiletest suite=incremental mode=incremental (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
---
Suite("src/test/debuginfo") not skipped for "bootstrap::test::Debuginfo" -- not in ["src/tools/tidy"]
Check compiletest suite=debuginfo mode=debuginfo (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)

running 116 tests
iiiii..i.....i..i...i..i.i.i..i..i..ii....i.i....ii..........iiii.........i.....i...i.......ii.i.ii. 100/116
....iiii.....ii.

 finished in 12.803
Suite("src/test/ui-fulldeps") not skipped for "bootstrap::test::UiFullDeps" -- not in ["src/tools/tidy"]
Uplifting stage1 rustc (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
---
 right: `2`', src/libstd/sync/mutex.rs:712:13
...thread '<unnamed>' panicked at 'test panic in inner thread to poison RwLock', src/libstd/sync/rwlock.rs:789:13
....thread '<unnamed>' panicked at 'test panic in inner thread to poison RwLock', src/libstd/sync/rwlock.rs:765:13
.thread '<unnamed>' panicked at 'explicit panic', src/libstd/sync/rwlock.rs:701:13
..thread '<unnamed>' panicked at 'explicit panic', src/libstd/sync/rwlock.rs:629:13
thread '<unnamed>' panicked at '.explicit panic', src/libstd/sync/rwlock.rs:641:13
thread '<unnamed>' panicked at 'explicit panic', src/libstd/sync/rwlock.rs:616:13
........................................... 700/769
..........................................thread '<unnamed>' panicked at 'explicit panic', src/libstd/thread/mod.rs:1576:37
..........thread '<unnamed>' panicked at 'Box<Any>', src/libstd/thread/mod.rs:1711:13
---
Set({"src/librustc_parse_format"}) not skipped for "bootstrap::test::CrateLibrustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_passes"}) not skipped for "bootstrap::test::CrateLibrustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_plugin_impl"}) not skipped for "bootstrap::test::CrateLibrustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_privacy"}) not skipped for "bootstrap::test::CrateLibrustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_query_system"}) not skipped for "bootstrap::test::CrateLibrustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_save_analysis"}) not skipped for "bootstrap::test::CrateLibrustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_serialize"}) not skipped for "bootstrap::test::CrateLibrustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_session"}) not skipped for "bootstrap::test::CrateLibrustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_span"}) not skipped for "bootstrap::test::CrateLibrustc" -- not in ["src/tools/tidy"]
---
For more information about this error, try `rustc --explain E0412`.
error: Could not document `core`.

Caused by:
  process didn't exit successfully: `/checkout/obj/build/bootstrap/debug/rustdoc --edition=2018 --crate-type lib --crate-name core src/libcore/lib.rs --target x86_64-unknown-linux-gnu -o /checkout/obj/build/x86_64-unknown-linux-gnu/stage2-std/x86_64-unknown-linux-gnu/doc --error-format=json --json=diagnostic-rendered-ansi --markdown-css rust.css --markdown-no-toc --generate-redirect-pages -Z unstable-options --resource-suffix 1.46.0 --index-page /checkout/src/doc/index.md -L dependency=/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-std/x86_64-unknown-linux-gnu/release/deps -L dependency=/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-std/release/deps` (exit code: 1)


command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "rustdoc" "--target" "x86_64-unknown-linux-gnu" "-Zbinary-dep-depinfo" "-j" "2" "--release" "--locked" "--color" "always" "--features" "panic-unwind backtrace compiler-builtins-c" "--manifest-path" "/checkout/src/libtest/Cargo.toml" "-p" "core" "--" "--markdown-css" "rust.css" "--markdown-no-toc" "--generate-redirect-pages" "-Z" "unstable-options" "--resource-suffix" "1.46.0" "--index-page" "/checkout/src/doc/index.md"


failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test --exclude src/tools/tidy
Build completed unsuccessfully in 1:17:38
Build completed unsuccessfully in 1:17:38
== clock drift check ==
  local time: Sat Jun 20 06:04:36 UTC 2020
  network time: Sat, 20 Jun 2020 06:04:36 GMT
== end clock drift check ==

##[error]Bash exited with code '1'.
##[section]Finishing: Run build
##[section]Starting: Checkout rust-lang/rust@refs/pull/73532/merge to s
Task         : Get sources
Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
Version      : 1.0.0
Author       : Microsoft
Author       : Microsoft
Help         : [More Information](https://go.microsoft.com/fwlink/?LinkId=798199)
==============================================================================
Cleaning any cached credential from repository: rust-lang/rust (GitHub)
##[section]Finishing: Checkout rust-lang/rust@refs/pull/73532/merge to s
Cleaning up task key
Start cleaning up orphan processes.
Terminate orphan process: pid (3479) (python)
##[section]Finishing: Finalize Job

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @rust-lang/infra. (Feature Requests)

@jyn514
Copy link
Member Author

jyn514 commented Jun 20, 2020

It seems like the DefId for the macro is being generated before this pass? Do you know where that happens? I think the AST pass needs to be run before the DefId is generated or else the DefId parent and the HirId parent won't match up.

@jyn514
Copy link
Member Author

jyn514 commented Jun 20, 2020

Ok yeah, DefIds are generated when macros are expanded:

let krate = sess.time("expand_crate", || ecx.monotonic_expander().expand_crate(krate));

which eventually calls
crate fn collect_definitions(

It makes sense to me that macros are expanded first (because they could generate new functions) but I'm not sure how DefIds fit in here. Maybe this pass can be moved after expansion but before the DefIds are generated?

@GuillaumeGomez
Copy link
Member

Sounds good to me.

@jyn514
Copy link
Member Author

jyn514 commented Jun 20, 2020

This doesn't do what I thought it did (the bug was not related to fake ids). If #73566 doesn't land then a modified version of #73103 should be used instead, not this.

@jyn514 jyn514 closed this Jun 20, 2020
@jyn514 jyn514 deleted the bouncer branch June 22, 2020 02:34
Manishearth added a commit to Manishearth/rust that referenced this pull request Jul 16, 2020
Don't run `everybody_loops` for rustdoc; instead ignore resolution errors

r? @eddyb
cc @petrochenkov, @GuillaumeGomez, @Manishearth, @ecstatic-morse, @marmeladema

~~Blocked on rust-lang#73743 Merged.
~~Blocked on crater run.~~ Crater popped up some ICEs ([now fixed](rust-lang#73566 (comment))). See [crater run](https://crater-reports.s3.amazonaws.com/pr-73566/index.html), [ICEs](rust-lang#73566 (comment)).
~~Blocked on rust-lang#74070 so that we don't make typeck_tables_of public when it shouldn't be.~~ Merged.

Closes rust-lang#71820, closes rust-lang#71104, closes rust-lang#65863.

## What is the motivation for this change?

As seen from a lengthy trail of PRs and issues (rust-lang#73532, rust-lang#73103, rust-lang#71820, rust-lang#71104), `everybody_loops` is causing bugs in rustdoc. The main issue is that it does not preserve the validity of the `DefId` tree, meaning that operations on DefIds may unexpectedly fail when called later. This is blocking intra-doc links (see rust-lang#73101).

This PR starts by removing `everybody_loops`, fixing rust-lang#71104 and rust-lang#71820. However, that brings back the bugs seen originally in rust-lang#43348: Since libstd documents items for all platforms, the function bodies sometimes do not type check. Here are the errors from documenting `libstd` with `everybody_loops` disabled and no other changes:

```rust
error[E0433]: failed to resolve: could not find `handle` in `sys`
  --> src/libstd/sys/windows/ext/process.rs:13:27
   |
13 |         let handle = sys::handle::Handle::new(handle as *mut _);
   |                           ^^^^^^ could not find `handle` in `sys`

error[E0425]: cannot find function `symlink_inner` in module `sys::fs`
   --> src/libstd/sys/windows/ext/fs.rs:544:14
    |
544 |     sys::fs::symlink_inner(src.as_ref(), dst.as_ref(), false)
    |              ^^^^^^^^^^^^^ not found in `sys::fs`

error[E0425]: cannot find function `symlink_inner` in module `sys::fs`
   --> src/libstd/sys/windows/ext/fs.rs:564:14
    |
564 |     sys::fs::symlink_inner(src.as_ref(), dst.as_ref(), true)
    |              ^^^^^^^^^^^^^ not found in `sys::fs`
```

## Why does this need changes to `rustc_resolve`?

Normally, this could be avoided by simply not calling the `typeck_item_bodies` pass. However, the errors above happen before type checking, in name resolution itself. Since name resolution is intermingled with macro expansion, and rustdoc needs expansion to happen before it knows all items to be documented, there needs to be someway to ignore _resolution_ errors in function bodies.

An alternative solution suggested by @petrochenkov was to not run `everybody_loops` on anything containing a nested `DefId`. This would solve some of the immediate issues, but isn't bullet-proof: the following functions still could not be documented if the items in the body failed to resolve:

- Functions containing a nested `DefId` (rust-lang#71104)
- ~~Functions returning `impl Trait` (rust-lang#43878 These ended up not resolving anyway with this PR.
- ~~`const fn`, because `loop {}` in `const fn` is unstable (rust-lang#43636 `const_loop` was just stabilized.

This also isn't exactly what rustdoc wants, which is to avoid looking at function bodies in the first place.

## What changes were made?

The hack implemented in this PR is to add an option to ignore all resolution errors in function bodies. This is enabled only for rustdoc. Since resolution errors are ignored, the MIR generated will be invalid, as can be seen in the following ICE:

```rust
error: internal compiler error: broken MIR in DefId(0:11 ~ doc_cfg[8787]::uses_target_feature[0]) ("return type"): bad type [type error]
  --> /home/joshua/src/rust/src/test/rustdoc/doc-cfg.rs:51:1
   |
51 | / pub unsafe fn uses_target_feature() {
52 | |     content::should::be::irrelevant();
53 | | }
   | |_^
```

Fortunately, rustdoc does not need to access MIR in order to generate documentation. Therefore this also removes the call to `analyze()` in `rustdoc::run_core`. This has the side effect of not generating all lints by default. Most lints are safe to ignore (does rustdoc really need to run liveness analysis?) but `missing_docs` in particular is disabled when it should not be. Re-running `missing_docs` specifically does not help, because it causes the typechecking pass to be run, bringing back the errors from rust-lang#24658:

```
error[E0599]: no method named `into_handle` found for struct `sys::unix::pipe::AnonPipe` in the current scope
  --> src/libstd/sys/windows/ext/process.rs:71:27
   |
71 |         self.into_inner().into_handle().into_raw() as *mut _
   |                           ^^^^^^^^^^^ method not found in `sys::unix::pipe::AnonPipe`
   |
```

Because of rust-lang#73743, we only run typeck on demand. So this only causes an issue for functions returning `impl Trait`, which were already special cased by `ReplaceFunctionWithBody`. However, it now considers `async fn f() -> T` to be considered `impl Future<Output = T>`, where before it was considered to have a concrete `T` type.

## How will this affect future changes to rustdoc?

- Any new changes to rustdoc will not be able to perform type checking without bringing back resolution errors in function bodies.
    + As a corollary, any new lints cannot require or perform type checking. In some cases this may require refactoring other parts of the compiler to perform type-checking only on-demand, see for example rust-lang#73743.
    + As a corollary, rustdoc can never again call `tcx.analysis()` unless this PR is reverted altogether.

## Current status

- ~~I am not yet sure how to bring back `missing_docs` without running typeck. @eddyb suggested allowing lints to opt-out of type-checking, which would probably be another rabbit hole.~~ The opt-out was implemented in rust-lang#73743. However, of the rustc lints, now _only_ missing_docs is run and no other lints: rust-lang#73566 (comment). We need a team decision on whether that's an acceptable tradeoff. Note that all rustdoc lints are still run (`intra_doc_link_resolution_failure`, etc). **UPDATE**: This was deemed acceptable in rust-lang#73566 (comment)
- ~~The implementation of optional errors in `rustc_resolve` is very brute force, it should probably be moved from `LateResolver` to `Resolver` to avoid duplicating the logic in many places.~~ I'm mostly happy with it now.

- This no longer allows errors in `async fn f() -> T`. This caused breakage in 50 crates out of a full crater run, all of which (that I looked at) didn't compile when run with rustc directly. In other words, it used to be that they could not be compiled but could still be documented; now they can't be documented either. This needs a decision from the rustdoc team on whether this is acceptable breakage. **UPDATE**: This was deemed acceptable in rust-lang#73566 (comment)
- ~~This makes `fn typeck_tables_of` in `rustc_typeck` public. This is not desired behavior, but needs the changes from rust-lang#74070 in order to be fixed.~~ Reverted.
Manishearth added a commit to Manishearth/rust that referenced this pull request Jul 16, 2020
Don't run `everybody_loops` for rustdoc; instead ignore resolution errors

r? @eddyb
cc @petrochenkov, @GuillaumeGomez, @Manishearth, @ecstatic-morse, @marmeladema

~~Blocked on rust-lang#73743 Merged.
~~Blocked on crater run.~~ Crater popped up some ICEs ([now fixed](rust-lang#73566 (comment))). See [crater run](https://crater-reports.s3.amazonaws.com/pr-73566/index.html), [ICEs](rust-lang#73566 (comment)).
~~Blocked on rust-lang#74070 so that we don't make typeck_tables_of public when it shouldn't be.~~ Merged.

Closes rust-lang#71820, closes rust-lang#71104, closes rust-lang#65863.

## What is the motivation for this change?

As seen from a lengthy trail of PRs and issues (rust-lang#73532, rust-lang#73103, rust-lang#71820, rust-lang#71104), `everybody_loops` is causing bugs in rustdoc. The main issue is that it does not preserve the validity of the `DefId` tree, meaning that operations on DefIds may unexpectedly fail when called later. This is blocking intra-doc links (see rust-lang#73101).

This PR starts by removing `everybody_loops`, fixing rust-lang#71104 and rust-lang#71820. However, that brings back the bugs seen originally in rust-lang#43348: Since libstd documents items for all platforms, the function bodies sometimes do not type check. Here are the errors from documenting `libstd` with `everybody_loops` disabled and no other changes:

```rust
error[E0433]: failed to resolve: could not find `handle` in `sys`
  --> src/libstd/sys/windows/ext/process.rs:13:27
   |
13 |         let handle = sys::handle::Handle::new(handle as *mut _);
   |                           ^^^^^^ could not find `handle` in `sys`

error[E0425]: cannot find function `symlink_inner` in module `sys::fs`
   --> src/libstd/sys/windows/ext/fs.rs:544:14
    |
544 |     sys::fs::symlink_inner(src.as_ref(), dst.as_ref(), false)
    |              ^^^^^^^^^^^^^ not found in `sys::fs`

error[E0425]: cannot find function `symlink_inner` in module `sys::fs`
   --> src/libstd/sys/windows/ext/fs.rs:564:14
    |
564 |     sys::fs::symlink_inner(src.as_ref(), dst.as_ref(), true)
    |              ^^^^^^^^^^^^^ not found in `sys::fs`
```

## Why does this need changes to `rustc_resolve`?

Normally, this could be avoided by simply not calling the `typeck_item_bodies` pass. However, the errors above happen before type checking, in name resolution itself. Since name resolution is intermingled with macro expansion, and rustdoc needs expansion to happen before it knows all items to be documented, there needs to be someway to ignore _resolution_ errors in function bodies.

An alternative solution suggested by @petrochenkov was to not run `everybody_loops` on anything containing a nested `DefId`. This would solve some of the immediate issues, but isn't bullet-proof: the following functions still could not be documented if the items in the body failed to resolve:

- Functions containing a nested `DefId` (rust-lang#71104)
- ~~Functions returning `impl Trait` (rust-lang#43878 These ended up not resolving anyway with this PR.
- ~~`const fn`, because `loop {}` in `const fn` is unstable (rust-lang#43636 `const_loop` was just stabilized.

This also isn't exactly what rustdoc wants, which is to avoid looking at function bodies in the first place.

## What changes were made?

The hack implemented in this PR is to add an option to ignore all resolution errors in function bodies. This is enabled only for rustdoc. Since resolution errors are ignored, the MIR generated will be invalid, as can be seen in the following ICE:

```rust
error: internal compiler error: broken MIR in DefId(0:11 ~ doc_cfg[8787]::uses_target_feature[0]) ("return type"): bad type [type error]
  --> /home/joshua/src/rust/src/test/rustdoc/doc-cfg.rs:51:1
   |
51 | / pub unsafe fn uses_target_feature() {
52 | |     content::should::be::irrelevant();
53 | | }
   | |_^
```

Fortunately, rustdoc does not need to access MIR in order to generate documentation. Therefore this also removes the call to `analyze()` in `rustdoc::run_core`. This has the side effect of not generating all lints by default. Most lints are safe to ignore (does rustdoc really need to run liveness analysis?) but `missing_docs` in particular is disabled when it should not be. Re-running `missing_docs` specifically does not help, because it causes the typechecking pass to be run, bringing back the errors from rust-lang#24658:

```
error[E0599]: no method named `into_handle` found for struct `sys::unix::pipe::AnonPipe` in the current scope
  --> src/libstd/sys/windows/ext/process.rs:71:27
   |
71 |         self.into_inner().into_handle().into_raw() as *mut _
   |                           ^^^^^^^^^^^ method not found in `sys::unix::pipe::AnonPipe`
   |
```

Because of rust-lang#73743, we only run typeck on demand. So this only causes an issue for functions returning `impl Trait`, which were already special cased by `ReplaceFunctionWithBody`. However, it now considers `async fn f() -> T` to be considered `impl Future<Output = T>`, where before it was considered to have a concrete `T` type.

## How will this affect future changes to rustdoc?

- Any new changes to rustdoc will not be able to perform type checking without bringing back resolution errors in function bodies.
    + As a corollary, any new lints cannot require or perform type checking. In some cases this may require refactoring other parts of the compiler to perform type-checking only on-demand, see for example rust-lang#73743.
    + As a corollary, rustdoc can never again call `tcx.analysis()` unless this PR is reverted altogether.

## Current status

- ~~I am not yet sure how to bring back `missing_docs` without running typeck. @eddyb suggested allowing lints to opt-out of type-checking, which would probably be another rabbit hole.~~ The opt-out was implemented in rust-lang#73743. However, of the rustc lints, now _only_ missing_docs is run and no other lints: rust-lang#73566 (comment). We need a team decision on whether that's an acceptable tradeoff. Note that all rustdoc lints are still run (`intra_doc_link_resolution_failure`, etc). **UPDATE**: This was deemed acceptable in rust-lang#73566 (comment)
- ~~The implementation of optional errors in `rustc_resolve` is very brute force, it should probably be moved from `LateResolver` to `Resolver` to avoid duplicating the logic in many places.~~ I'm mostly happy with it now.

- This no longer allows errors in `async fn f() -> T`. This caused breakage in 50 crates out of a full crater run, all of which (that I looked at) didn't compile when run with rustc directly. In other words, it used to be that they could not be compiled but could still be documented; now they can't be documented either. This needs a decision from the rustdoc team on whether this is acceptable breakage. **UPDATE**: This was deemed acceptable in rust-lang#73566 (comment)
- ~~This makes `fn typeck_tables_of` in `rustc_typeck` public. This is not desired behavior, but needs the changes from rust-lang#74070 in order to be fixed.~~ Reverted.
Manishearth added a commit to Manishearth/rust that referenced this pull request Jul 16, 2020
Don't run `everybody_loops` for rustdoc; instead ignore resolution errors

r? @eddyb
cc @petrochenkov, @GuillaumeGomez, @Manishearth, @ecstatic-morse, @marmeladema

~~Blocked on rust-lang#73743 Merged.
~~Blocked on crater run.~~ Crater popped up some ICEs ([now fixed](rust-lang#73566 (comment))). See [crater run](https://crater-reports.s3.amazonaws.com/pr-73566/index.html), [ICEs](rust-lang#73566 (comment)).
~~Blocked on rust-lang#74070 so that we don't make typeck_tables_of public when it shouldn't be.~~ Merged.

Closes rust-lang#71820, closes rust-lang#71104, closes rust-lang#65863.

## What is the motivation for this change?

As seen from a lengthy trail of PRs and issues (rust-lang#73532, rust-lang#73103, rust-lang#71820, rust-lang#71104), `everybody_loops` is causing bugs in rustdoc. The main issue is that it does not preserve the validity of the `DefId` tree, meaning that operations on DefIds may unexpectedly fail when called later. This is blocking intra-doc links (see rust-lang#73101).

This PR starts by removing `everybody_loops`, fixing rust-lang#71104 and rust-lang#71820. However, that brings back the bugs seen originally in rust-lang#43348: Since libstd documents items for all platforms, the function bodies sometimes do not type check. Here are the errors from documenting `libstd` with `everybody_loops` disabled and no other changes:

```rust
error[E0433]: failed to resolve: could not find `handle` in `sys`
  --> src/libstd/sys/windows/ext/process.rs:13:27
   |
13 |         let handle = sys::handle::Handle::new(handle as *mut _);
   |                           ^^^^^^ could not find `handle` in `sys`

error[E0425]: cannot find function `symlink_inner` in module `sys::fs`
   --> src/libstd/sys/windows/ext/fs.rs:544:14
    |
544 |     sys::fs::symlink_inner(src.as_ref(), dst.as_ref(), false)
    |              ^^^^^^^^^^^^^ not found in `sys::fs`

error[E0425]: cannot find function `symlink_inner` in module `sys::fs`
   --> src/libstd/sys/windows/ext/fs.rs:564:14
    |
564 |     sys::fs::symlink_inner(src.as_ref(), dst.as_ref(), true)
    |              ^^^^^^^^^^^^^ not found in `sys::fs`
```

## Why does this need changes to `rustc_resolve`?

Normally, this could be avoided by simply not calling the `typeck_item_bodies` pass. However, the errors above happen before type checking, in name resolution itself. Since name resolution is intermingled with macro expansion, and rustdoc needs expansion to happen before it knows all items to be documented, there needs to be someway to ignore _resolution_ errors in function bodies.

An alternative solution suggested by @petrochenkov was to not run `everybody_loops` on anything containing a nested `DefId`. This would solve some of the immediate issues, but isn't bullet-proof: the following functions still could not be documented if the items in the body failed to resolve:

- Functions containing a nested `DefId` (rust-lang#71104)
- ~~Functions returning `impl Trait` (rust-lang#43878 These ended up not resolving anyway with this PR.
- ~~`const fn`, because `loop {}` in `const fn` is unstable (rust-lang#43636 `const_loop` was just stabilized.

This also isn't exactly what rustdoc wants, which is to avoid looking at function bodies in the first place.

## What changes were made?

The hack implemented in this PR is to add an option to ignore all resolution errors in function bodies. This is enabled only for rustdoc. Since resolution errors are ignored, the MIR generated will be invalid, as can be seen in the following ICE:

```rust
error: internal compiler error: broken MIR in DefId(0:11 ~ doc_cfg[8787]::uses_target_feature[0]) ("return type"): bad type [type error]
  --> /home/joshua/src/rust/src/test/rustdoc/doc-cfg.rs:51:1
   |
51 | / pub unsafe fn uses_target_feature() {
52 | |     content::should::be::irrelevant();
53 | | }
   | |_^
```

Fortunately, rustdoc does not need to access MIR in order to generate documentation. Therefore this also removes the call to `analyze()` in `rustdoc::run_core`. This has the side effect of not generating all lints by default. Most lints are safe to ignore (does rustdoc really need to run liveness analysis?) but `missing_docs` in particular is disabled when it should not be. Re-running `missing_docs` specifically does not help, because it causes the typechecking pass to be run, bringing back the errors from rust-lang#24658:

```
error[E0599]: no method named `into_handle` found for struct `sys::unix::pipe::AnonPipe` in the current scope
  --> src/libstd/sys/windows/ext/process.rs:71:27
   |
71 |         self.into_inner().into_handle().into_raw() as *mut _
   |                           ^^^^^^^^^^^ method not found in `sys::unix::pipe::AnonPipe`
   |
```

Because of rust-lang#73743, we only run typeck on demand. So this only causes an issue for functions returning `impl Trait`, which were already special cased by `ReplaceFunctionWithBody`. However, it now considers `async fn f() -> T` to be considered `impl Future<Output = T>`, where before it was considered to have a concrete `T` type.

## How will this affect future changes to rustdoc?

- Any new changes to rustdoc will not be able to perform type checking without bringing back resolution errors in function bodies.
    + As a corollary, any new lints cannot require or perform type checking. In some cases this may require refactoring other parts of the compiler to perform type-checking only on-demand, see for example rust-lang#73743.
    + As a corollary, rustdoc can never again call `tcx.analysis()` unless this PR is reverted altogether.

## Current status

- ~~I am not yet sure how to bring back `missing_docs` without running typeck. @eddyb suggested allowing lints to opt-out of type-checking, which would probably be another rabbit hole.~~ The opt-out was implemented in rust-lang#73743. However, of the rustc lints, now _only_ missing_docs is run and no other lints: rust-lang#73566 (comment). We need a team decision on whether that's an acceptable tradeoff. Note that all rustdoc lints are still run (`intra_doc_link_resolution_failure`, etc). **UPDATE**: This was deemed acceptable in rust-lang#73566 (comment)
- ~~The implementation of optional errors in `rustc_resolve` is very brute force, it should probably be moved from `LateResolver` to `Resolver` to avoid duplicating the logic in many places.~~ I'm mostly happy with it now.

- This no longer allows errors in `async fn f() -> T`. This caused breakage in 50 crates out of a full crater run, all of which (that I looked at) didn't compile when run with rustc directly. In other words, it used to be that they could not be compiled but could still be documented; now they can't be documented either. This needs a decision from the rustdoc team on whether this is acceptable breakage. **UPDATE**: This was deemed acceptable in rust-lang#73566 (comment)
- ~~This makes `fn typeck_tables_of` in `rustc_typeck` public. This is not desired behavior, but needs the changes from rust-lang#74070 in order to be fixed.~~ Reverted.
Manishearth added a commit to Manishearth/rust that referenced this pull request Jul 16, 2020
Don't run `everybody_loops` for rustdoc; instead ignore resolution errors

r? @eddyb
cc @petrochenkov, @GuillaumeGomez, @Manishearth, @ecstatic-morse, @marmeladema

~~Blocked on rust-lang#73743 Merged.
~~Blocked on crater run.~~ Crater popped up some ICEs ([now fixed](rust-lang#73566 (comment))). See [crater run](https://crater-reports.s3.amazonaws.com/pr-73566/index.html), [ICEs](rust-lang#73566 (comment)).
~~Blocked on rust-lang#74070 so that we don't make typeck_tables_of public when it shouldn't be.~~ Merged.

Closes rust-lang#71820, closes rust-lang#71104, closes rust-lang#65863.

## What is the motivation for this change?

As seen from a lengthy trail of PRs and issues (rust-lang#73532, rust-lang#73103, rust-lang#71820, rust-lang#71104), `everybody_loops` is causing bugs in rustdoc. The main issue is that it does not preserve the validity of the `DefId` tree, meaning that operations on DefIds may unexpectedly fail when called later. This is blocking intra-doc links (see rust-lang#73101).

This PR starts by removing `everybody_loops`, fixing rust-lang#71104 and rust-lang#71820. However, that brings back the bugs seen originally in rust-lang#43348: Since libstd documents items for all platforms, the function bodies sometimes do not type check. Here are the errors from documenting `libstd` with `everybody_loops` disabled and no other changes:

```rust
error[E0433]: failed to resolve: could not find `handle` in `sys`
  --> src/libstd/sys/windows/ext/process.rs:13:27
   |
13 |         let handle = sys::handle::Handle::new(handle as *mut _);
   |                           ^^^^^^ could not find `handle` in `sys`

error[E0425]: cannot find function `symlink_inner` in module `sys::fs`
   --> src/libstd/sys/windows/ext/fs.rs:544:14
    |
544 |     sys::fs::symlink_inner(src.as_ref(), dst.as_ref(), false)
    |              ^^^^^^^^^^^^^ not found in `sys::fs`

error[E0425]: cannot find function `symlink_inner` in module `sys::fs`
   --> src/libstd/sys/windows/ext/fs.rs:564:14
    |
564 |     sys::fs::symlink_inner(src.as_ref(), dst.as_ref(), true)
    |              ^^^^^^^^^^^^^ not found in `sys::fs`
```

## Why does this need changes to `rustc_resolve`?

Normally, this could be avoided by simply not calling the `typeck_item_bodies` pass. However, the errors above happen before type checking, in name resolution itself. Since name resolution is intermingled with macro expansion, and rustdoc needs expansion to happen before it knows all items to be documented, there needs to be someway to ignore _resolution_ errors in function bodies.

An alternative solution suggested by @petrochenkov was to not run `everybody_loops` on anything containing a nested `DefId`. This would solve some of the immediate issues, but isn't bullet-proof: the following functions still could not be documented if the items in the body failed to resolve:

- Functions containing a nested `DefId` (rust-lang#71104)
- ~~Functions returning `impl Trait` (rust-lang#43878 These ended up not resolving anyway with this PR.
- ~~`const fn`, because `loop {}` in `const fn` is unstable (rust-lang#43636 `const_loop` was just stabilized.

This also isn't exactly what rustdoc wants, which is to avoid looking at function bodies in the first place.

## What changes were made?

The hack implemented in this PR is to add an option to ignore all resolution errors in function bodies. This is enabled only for rustdoc. Since resolution errors are ignored, the MIR generated will be invalid, as can be seen in the following ICE:

```rust
error: internal compiler error: broken MIR in DefId(0:11 ~ doc_cfg[8787]::uses_target_feature[0]) ("return type"): bad type [type error]
  --> /home/joshua/src/rust/src/test/rustdoc/doc-cfg.rs:51:1
   |
51 | / pub unsafe fn uses_target_feature() {
52 | |     content::should::be::irrelevant();
53 | | }
   | |_^
```

Fortunately, rustdoc does not need to access MIR in order to generate documentation. Therefore this also removes the call to `analyze()` in `rustdoc::run_core`. This has the side effect of not generating all lints by default. Most lints are safe to ignore (does rustdoc really need to run liveness analysis?) but `missing_docs` in particular is disabled when it should not be. Re-running `missing_docs` specifically does not help, because it causes the typechecking pass to be run, bringing back the errors from rust-lang#24658:

```
error[E0599]: no method named `into_handle` found for struct `sys::unix::pipe::AnonPipe` in the current scope
  --> src/libstd/sys/windows/ext/process.rs:71:27
   |
71 |         self.into_inner().into_handle().into_raw() as *mut _
   |                           ^^^^^^^^^^^ method not found in `sys::unix::pipe::AnonPipe`
   |
```

Because of rust-lang#73743, we only run typeck on demand. So this only causes an issue for functions returning `impl Trait`, which were already special cased by `ReplaceFunctionWithBody`. However, it now considers `async fn f() -> T` to be considered `impl Future<Output = T>`, where before it was considered to have a concrete `T` type.

## How will this affect future changes to rustdoc?

- Any new changes to rustdoc will not be able to perform type checking without bringing back resolution errors in function bodies.
    + As a corollary, any new lints cannot require or perform type checking. In some cases this may require refactoring other parts of the compiler to perform type-checking only on-demand, see for example rust-lang#73743.
    + As a corollary, rustdoc can never again call `tcx.analysis()` unless this PR is reverted altogether.

## Current status

- ~~I am not yet sure how to bring back `missing_docs` without running typeck. @eddyb suggested allowing lints to opt-out of type-checking, which would probably be another rabbit hole.~~ The opt-out was implemented in rust-lang#73743. However, of the rustc lints, now _only_ missing_docs is run and no other lints: rust-lang#73566 (comment). We need a team decision on whether that's an acceptable tradeoff. Note that all rustdoc lints are still run (`intra_doc_link_resolution_failure`, etc). **UPDATE**: This was deemed acceptable in rust-lang#73566 (comment)
- ~~The implementation of optional errors in `rustc_resolve` is very brute force, it should probably be moved from `LateResolver` to `Resolver` to avoid duplicating the logic in many places.~~ I'm mostly happy with it now.

- This no longer allows errors in `async fn f() -> T`. This caused breakage in 50 crates out of a full crater run, all of which (that I looked at) didn't compile when run with rustc directly. In other words, it used to be that they could not be compiled but could still be documented; now they can't be documented either. This needs a decision from the rustdoc team on whether this is acceptable breakage. **UPDATE**: This was deemed acceptable in rust-lang#73566 (comment)
- ~~This makes `fn typeck_tables_of` in `rustc_typeck` public. This is not desired behavior, but needs the changes from rust-lang#74070 in order to be fixed.~~ Reverted.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
S-waiting-on-review Status: Awaiting review from the assignee but also interested parties.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Invalid lowering of macro in closure Some NodeId/LocalDefId don't have a corresponding HirId
4 participants