Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rustdoc: Replaces fn main search and extern crate search with proper parsing during doctests. #54861

Merged
merged 8 commits into from
Nov 4, 2018

Conversation

repnop
Copy link
Contributor

@repnop repnop commented Oct 5, 2018

Fixes #21299.
Fixes #33731.

Let me know if there's any additional changes you'd like made!

@rust-highfive

This comment has been minimized.

@rust-highfive

This comment has been minimized.

@rust-highfive

This comment has been minimized.

@TimNN TimNN added T-rustdoc Relevant to the rustdoc team, which will review and decide on the PR/issue. S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Oct 9, 2018
@TimNN
Copy link
Contributor

TimNN commented Oct 9, 2018

Thanks for your PR! You should hear from someone from @QuietMisdreavus or someone else on the Rustdoc team soon.

r? @QuietMisdreavus

@GuillaumeGomez
Copy link
Member

Wo, nice one!

@repnop
Copy link
Contributor Author

repnop commented Oct 17, 2018

Should hopefully have time either towards the end of the week or the weekend to get this PR fixed up and ready to go 👌

@rust-highfive
Copy link
Collaborator

The job x86_64-gnu-llvm-5.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
[00:43:49]     |              ^^^^^^^^^^^^^^^ cannot be resolved, ignoring
[00:43:49]     |
[00:43:49]     = help: to escape `[` and `]` characters, just add '\' before them like `\[` or `\]`
[00:43:49] 
[00:43:51] error: expected item after attributes
[00:43:51]  --> <anon>:1:31
[00:43:51]   |
[00:43:51] 1 | #[feature(exhaustive_patterns)]
[00:43:51] 
[00:43:51] 
[00:43:51] error: macros that expand to items must either be surrounded with braces or followed by a semicolon
[00:43:51]  --> <anon>:2:15
[00:43:51]   |
[00:43:51] 2 | compile_error!("Either feature \"foo\" or \"bar\" must be enabled for this crate.")
[00:43:51] 
[00:43:51] 
[00:43:51] error: unexpected token: `...`
[00:43:51]   --> <anon>:10:26
[00:43:51]    |
[00:43:51] 10 |         _mm256_add_epi64(...);
[00:43:51]    |                          ^^^
[00:43:51] help: use `..` for an exclusive range
[00:43:51]    |
[00:43:51] 10 |         _mm256_add_epi64(..);
[00:43:51]    |                          ^^
[00:43:51] help: or `..=` for an inclusive range
[00:43:51]    |
[00:43:51] 10 |         _mm256_add_epi64(..=);
[00:43:51] 
[00:43:51] 
[00:43:51] error[E0586]: inclusive range with no end
[00:43:51]   --> <anon>:10:29
[00:43:51]    |
[00:43:51] 10 |         _mm256_add_epi64(...);
[00:43:51]    |
[00:43:51]    |
[00:43:51]    = help: inclusive ranges must be bounded at the end (`..=b` or `a..=b`)
[00:43:51] 
[00:43:51] error: unexpected token: `...`
[00:43:51]   --> <anon>:20:22
[00:43:51]    |
[00:43:51] 20 |     _mm256_add_epi64(...);
[00:43:51]    |                      ^^^
[00:43:51] help: use `..` for an exclusive range
[00:43:51]    |
[00:43:51] 20 |     _mm256_add_epi64(..);
[00:43:51]    |                      ^^
[00:43:51] help: or `..=` for an inclusive range
[00:43:51]    |
[00:43:51] 20 |     _mm256_add_epi64(..=);
[00:43:51] 
[00:43:51] 
[00:43:51] error[E0586]: inclusive range with no end
[00:43:51]   --> <anon>:20:25
[00:43:51]    |
[00:43:51] 20 |     _mm256_add_epi64(...);
[00:43:51]    |
[00:43:51]    |
[00:43:51]    = help: inclusive ranges must be bounded at the end (`..=b` or `a..=b`)
[00:43:53]     Finished release [optimized] target(s) in 50.60s
[00:43:53] Documenting stage2 test (x86_64-unknown-linux-gnu)
[00:43:54]     Checking term v0.0.0 (/checkout/src/libterm)
[00:43:54]     Checking getopts v0.2.17
---
[00:44:34] travis_fold:end:stage2-error_index_generator

[00:44:34] travis_time:end:stage2-error_index_generator:start=1540064024085535125,finish=1540064027269330388,duration=3183795263

[00:44:34] error: only foreign functions are allowed to be variadic
[00:44:34]  --> <anon>:5:15
[00:44:34]   |
[00:44:34] 5 | fn foo(x: u8, ...) {}
[00:44:34] 
[00:44:34] 
[00:44:34] error[E0178]: expected a path on the left-hand side of `+`, not `&'a Foo`
[00:44:34]  --> <anon>:4:8
[00:44:34]   |
[00:44:34] 4 |     w: &'a Foo + Copy,   // error, use &'a (Foo + Copy)
[00:44:34]   |        ^^^^^^^^^^^^^^ help: try adding parentheses: `&'a (Foo + Copy)`
[00:44:34] 
[00:44:34] error[E0178]: expected a path on the left-hand side of `+`, not `&'a Foo`
[00:44:34]  --> <anon>:5:8
[00:44:34]   |
[00:44:34] 5 |     x: &'a Foo + 'a,     // error, use &'a (Foo + 'a)
[00:44:34]   |        ^^^^^^^^^^^^ help: try adding parentheses: `&'a (Foo + 'a)`
[00:44:34] 
[00:44:34] error[E0178]: expected a path on the left-hand side of `+`, not `&'a mut Foo`
[00:44:34]  --> <anon>:6:8
[00:44:34]   |
[00:44:34] 6 |     y: &'a mut Foo + 'a, // error, use &'a mut (Foo + 'a)
[00:44:34]   |        ^^^^^^^^^^^^^^^^ help: try adding parentheses: `&'a mut (Foo + 'a)`
[00:44:34] 
[00:44:34] error[E0178]: expected a path on the left-hand side of `+`, not `fn() -> Foo`
[00:44:34]  --> <anon>:7:8
[00:44:34]   |
[00:44:34] 7 |     z: fn() -> Foo + 'a, // error, use fn() -> (Foo + 'a)
[00:44:34]   |        ^^^^^^^^^^^^^^^^ perhaps you forgot parentheses?
[00:44:34] 
[00:44:34] error: unexpected close delimiter: `}`
[00:44:34]  --> <anon>:5:1
[00:44:34] 5 | }
[00:44:34] 5 | }
[00:44:34]   | ^ unexpected close delimiter
[00:44:34] 
[00:44:34] 
[00:44:34] 
[00:44:34] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-tools-bin/error_index_generator" "html" "/checkout/obj/build/x86_64-unknown-linux-gnu/doc/error-index.html"
[00:44:34] 
[00:44:34] 
[00:44:34] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap doc
[00:44:34] Build completed unsuccessfully in 0:05:24
[00:44:34] Build completed unsuccessfully in 0:05:24
[00:44:34] Makefile:28: recipe for target 'all' failed
[00:44:34] make: *** [all] Error 1

The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:17be19a8
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@TimNN
Copy link
Contributor

TimNN commented Oct 30, 2018

Ping from triage @QuietMisdreavus: This PR requires your review.

@QuietMisdreavus
Copy link
Member

Looks like the parsing errors are still being printed, though it's not causing rustdoc to fail:

[00:43:51] error: expected item after attributes
[00:43:51]  --> <anon>:1:31
[00:43:51]   |
[00:43:51] 1 | #[feature(exhaustive_patterns)]
[00:43:51]   |                               ^
[00:43:51] 
[00:43:51] error: macros that expand to items must either be surrounded with braces or followed by a semicolon
[00:43:51]  --> <anon>:2:15
[00:43:51]   |
[00:43:51] 2 | compile_error!("Either feature \"foo\" or \"bar\" must be enabled for this crate.")
[00:43:51]   |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[00:43:51] 
[00:43:51] error: unexpected token: `...`
[00:43:51]   --> <anon>:10:26
[00:43:51]    |
[00:43:51] 10 |         _mm256_add_epi64(...);
[00:43:51]    |                          ^^^
[00:43:51] help: use `..` for an exclusive range
[00:43:51]    |
[00:43:51] 10 |         _mm256_add_epi64(..);
[00:43:51]    |                          ^^
[00:43:51] help: or `..=` for an inclusive range
[00:43:51]    |
[00:43:51] 10 |         _mm256_add_epi64(..=);
[00:43:51]    |                          ^^^
[00:43:51] 
[00:43:51] error[E0586]: inclusive range with no end
[00:43:51]   --> <anon>:10:29
[00:43:51]    |
[00:43:51] 10 |         _mm256_add_epi64(...);
[00:43:51]    |                             ^
[00:43:51]    |
[00:43:51]    = help: inclusive ranges must be bounded at the end (`..=b` or `a..=b`)
[00:43:51] 
[00:43:51] error: unexpected token: `...`
[00:43:51]   --> <anon>:20:22
[00:43:51]    |
[00:43:51] 20 |     _mm256_add_epi64(...);
[00:43:51]    |                      ^^^
[00:43:51] help: use `..` for an exclusive range
[00:43:51]    |
[00:43:51] 20 |     _mm256_add_epi64(..);
[00:43:51]    |                      ^^
[00:43:51] help: or `..=` for an inclusive range
[00:43:51]    |
[00:43:51] 20 |     _mm256_add_epi64(..=);
[00:43:51]    |                      ^^^
[00:43:51] 
[00:43:51] error[E0586]: inclusive range with no end
[00:43:51]   --> <anon>:20:25
[00:43:51]    |
[00:43:51] 20 |     _mm256_add_epi64(...);
[00:43:51]    |                         ^
[00:43:51]    |
[00:43:51]    = help: inclusive ranges must be bounded at the end (`..=b` or `a..=b`)

However, it looks like what actually caused the failure was the error index generator? There's a new batch of errors at the end of the travis log:

[00:44:34] error: only foreign functions are allowed to be variadic
[00:44:34]  --> <anon>:5:15
[00:44:34]   |
[00:44:34] 5 | fn foo(x: u8, ...) {}
[00:44:34]   |               ^^^
[00:44:34] 
[00:44:34] error[E0178]: expected a path on the left-hand side of `+`, not `&'a Foo`
[00:44:34]  --> <anon>:4:8
[00:44:34]   |
[00:44:34] 4 |     w: &'a Foo + Copy,   // error, use &'a (Foo + Copy)
[00:44:34]   |        ^^^^^^^^^^^^^^ help: try adding parentheses: `&'a (Foo + Copy)`
[00:44:34] 
[00:44:34] error[E0178]: expected a path on the left-hand side of `+`, not `&'a Foo`
[00:44:34]  --> <anon>:5:8
[00:44:34]   |
[00:44:34] 5 |     x: &'a Foo + 'a,     // error, use &'a (Foo + 'a)
[00:44:34]   |        ^^^^^^^^^^^^ help: try adding parentheses: `&'a (Foo + 'a)`
[00:44:34] 
[00:44:34] error[E0178]: expected a path on the left-hand side of `+`, not `&'a mut Foo`
[00:44:34]  --> <anon>:6:8
[00:44:34]   |
[00:44:34] 6 |     y: &'a mut Foo + 'a, // error, use &'a mut (Foo + 'a)
[00:44:34]   |        ^^^^^^^^^^^^^^^^ help: try adding parentheses: `&'a mut (Foo + 'a)`
[00:44:34] 
[00:44:34] error[E0178]: expected a path on the left-hand side of `+`, not `fn() -> Foo`
[00:44:34]  --> <anon>:7:8
[00:44:34]   |
[00:44:34] 7 |     z: fn() -> Foo + 'a, // error, use fn() -> (Foo + 'a)
[00:44:34]   |        ^^^^^^^^^^^^^^^^ perhaps you forgot parentheses?
[00:44:34] 
[00:44:34] error: unexpected close delimiter: `}`
[00:44:34]  --> <anon>:5:1
[00:44:34]   |
[00:44:34] 5 | }
[00:44:34]   | ^ unexpected close delimiter

I need to look closer to see why the error-index generator is running into problems, but that's at least the source of the current travis failure.

@repnop
Copy link
Contributor Author

repnop commented Oct 31, 2018

Odd that the errors are still being printed, I didn't experience that doing a local doc build. But yeah, those hard errors are what have caused me to get kind of stuck :( I tried a bunch of different things locally, from how I was handling the errors, to trying different ways of cancelling -- all to no avail. If you have any hints on the cause I should hopefully have time this weekend to fix it up and then maybe, just maybe, the PR will finally build 😄

@QuietMisdreavus
Copy link
Member

The reason the error-index generator is running through your code is because it uses rustdoc's Markdown parsing code to output HTML, and the Markdown parsing uses the test generator to create playground links for code samples.

I also looked into libsyntax, and i think i know why it's still emitting the errors: the parser code doesn't bubble up every error it creates. For example, here's the spot in the parser that corresponds to the first error from the error-index:

p.struct_span_err(
span,
"only foreign functions are allowed to be variadic"
).emit();

Since the parser considers it an error but wants to continue parsing the function to emit any other errors, it immediately emits the error and carries on. To truly silence errors from the parser, it looks like we'll need to go through this parsing code and figure out a way to silence them, but only for this one location.

I also found why it failed entirely: any errors that come out of the conversion to a TokenStream (done during the creation of the Parser) that are immediately thrown as a fatal error. This includes the "unexpected close delimiter" at the end of the error index's errors. I'm having trouble finding the specific example that is causing the issue, though, since so many samples are supposed to fail compiling. There's also the possibility that something about the partition_source implementation is causing malformed code to be sent to the parser. I'll try digging a little more.

@QuietMisdreavus
Copy link
Member

Update: I found it! I had to wire env_logger into the error-index generator so i could print logs from rustdoc, but i eventually found out that the program that was failing looked like this:

image

The comment on the crate line absorbs the opening delimiter for the module that the test defines, because there was no line break between them, even though that line break existed in the original. I'm rebuilding now, but you should add crates.push_str("\n"); to your addition to partition_source, so that this can properly parse.

That just leaves the errors being printed. I'm investigating using an error emitter that sends information to an io::sink() instead of stdout, so that the messages get ignored entirely. It requires a lot more setup, and it'll probably require adding a method to libsyntax so that we can parse a TokenStream without printing or emitting an error from its first token.

@QuietMisdreavus
Copy link
Member

I wound up writing the fix in myself, so i'm going to swap in a new reviewer:

r? @GuillaumeGomez

I'd also like someone from @rust-lang/compiler to take a look since i'm adding a few new methods in libsyntax and i want to make sure i'm not stepping on anyone's toes.

@@ -174,14 +174,25 @@ pub fn parse_stream_from_source_str(name: FileName, source: String, sess: &Parse
source_file_to_stream(sess, sess.source_map().new_source_file(name, source), override_span)
}

// Create a new parser from a source string
/// Create a new parser from a source string
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this use maybe_new_parser_from_source_str and emit the buffered errors? That would also allow removing source_file_to_parser, I think.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How deep do you want me to go? Should i reimplement each existing function in terms of its new maybe_ variant?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you have the time, that'd be great. I just want to minimize code duplication. Whenever I add a fallible version of an existing method, I rewrite the callees if under a dozen, or I rewrite the infallible version to call the fallible version.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense. I've got the change written locally, i'll push once i finish my current build.

@@ -175,6 +175,16 @@ impl<'a> StringReader<'a> {
self.fatal_errs.clear();
}

pub fn buffer_fatal_errors(&mut self) -> Vec<Diagnostic> {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was this was the only API addition you needed?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This and the new_or_buffered_errs constructor. I needed a way to "handle" the errors without prematurely emitting them (in this case, to cancel them), and this felt like a natural way to get the content of self.fatal_errs without just exposing the Vec directly. Since both new and new_without_err emitted the errors directly, i added this instead.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not against it, I'm surprised at how little you needed to add :)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup! I just needed a version that wouldn't immediately crash the process if it ran into a problem in the first token. One thing i will ask is whether the current implementation with buffer() is appropriate, since plain Diagnostics don't have the destructor bomb that full DiagnosticBuilders have. (Since they need a Handler to be emitted, they don't have the same ability to forcibly print themselves.) The alternative would be to swap out the Vec entirely and return the full DiagnosticBuilder, and since the ParseSess isn't local the lifetimes would work out.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rust-highfive

This comment has been minimized.

@QuietMisdreavus
Copy link
Member

Whoops, wrote new_parser_from_source_str wrong. I amended the last commit and force-pushed.

@estebank
Copy link
Contributor

estebank commented Nov 2, 2018

It seems to have failed again with the same error

[01:32:52] error: expected identifier, found keyword `return`
[01:32:52]  --> bogofile:1:13
[01:32:52]   |
[01:32:52] 1 | ::abc::def::return
[01:32:52]   |             ^^^^^^ expected identifier, found keyword
[01:32:52] 
[01:32:52] ..........................F...........................................................
[01:32:52] failures:
[01:32:52] 
[01:32:52] ---- parse::tests::out_of_line_mod stdout ----
[01:32:52] thread 'parse::tests::out_of_line_mod' panicked at 'called `Result::unwrap()` on an `Err` value: Diagnostic { level: Error, message: [("file not found for module `this_does_not_exist`", NoStyle)], code: Some(Error("E0583")), span: MultiSpan { primary_spans: [Span { lo: BytePos(24), hi: BytePos(43), ctxt: #0 }], span_labels: [] }, children: [SubDiagnostic { level: Help, message: [("name the file either this_does_not_exist.rs or this_does_not_exist/mod.rs inside the directory \"foo\"", NoStyle)], span: MultiSpan { primary_spans: [], span_labels: [] }, render_span: None }], suggestions: [] }', libcore/result.rs:1009:5
[01:32:52] note: Run with `RUST_BACKTRACE=1` for a backtrace.
[01:32:52] 
[01:32:52] 
[01:32:52] failures:
[01:32:52]     parse::tests::out_of_line_mod
[01:32:52] 
[01:32:52] test result: FAILED. 85 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out

@QuietMisdreavus
Copy link
Member

That message about ::abc::def::return is normal, that's printed even on successful runs.

But the panic message... that passes locally, so i have no idea what it's running into! o_O Let me try to run a clean build and see if it still fails.

@QuietMisdreavus
Copy link
Member

Oh hey, my bluff about just forcing travis to try again seems to have worked! CI is green now.

@estebank
Copy link
Contributor

estebank commented Nov 3, 2018

@bors r+

@bors
Copy link
Contributor

bors commented Nov 3, 2018

📌 Commit 014c8c4 has been approved by estebank

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Nov 3, 2018
@bors
Copy link
Contributor

bors commented Nov 4, 2018

⌛ Testing commit 014c8c4 with merge 6d69fe7...

bors added a commit that referenced this pull request Nov 4, 2018
rustdoc: Replaces fn main search and extern crate search with proper parsing during doctests.

Fixes #21299.
Fixes #33731.

Let me know if there's any additional changes you'd like made!
@bors
Copy link
Contributor

bors commented Nov 4, 2018

☀️ Test successful - status-appveyor, status-travis
Approved by: estebank
Pushing 6d69fe7 to master...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-rustdoc Relevant to the rustdoc team, which will review and decide on the PR/issue.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants