Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Filter away test annotations from UI test output #59044

Merged
merged 4 commits into from
Mar 12, 2019

Conversation

petrochenkov
Copy link
Contributor

If you worked with UI tests for some time you could notice one issue affecting their readability and also readability of diffs when the tests change.
Look at the output of this test.

fn main() {
    let 1 = 2; //~ ERROR refutable pattern in local binding
}
error[E0005]: refutable pattern in local binding: `-2147483648i32..=0i32` not covered
 --> src/main.rs:2:9
  |
2 |     let 1 = 2; //~ ERROR refutable pattern in local binding
  |         ^ pattern `-2147483648i32..=0i32` not covered

error: aborting due to previous error

For more information about this error, try `rustc --explain E0005`.

You can see that the "refutable pattern in local binding" is duplicated.
One instance is the actual error, and the second instance is the expected error annotation.
This annotation is useful in the test input, but in the output it clutters the text and makes it harder to see what text refers to actual errors and what is just comments, especially if there are many errors in a single test file.

@estebank reported using the next trick to avoid the clutter

fn main() {
    let 1 = 2;
    //~^ ERROR refutable pattern in local binding
}
error[E0005]: refutable pattern in local binding: `-2147483648i32..=0i32` not covered
 --> src/main.rs:2:9
  |
2 |     let 1 = 2;
  |         ^ pattern `-2147483648i32..=0i32` not covered

error: aborting due to previous error

For more information about this error, try `rustc --explain E0005`.

, i.e. using //~^ and placing the annotation one line below will remove the annotation from the output.

However, this doesn't always works (consider errors with multi-line spans), and shouldn't be necessary in general!
compiletest could automatically filter away its own annotations from the output instead.
This is exactly what this PR does.

r? @davidtwco

@rust-highfive rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Mar 9, 2019
@petrochenkov
Copy link
Contributor Author

cc @rust-lang/compiler

@rust-highfive
Copy link
Collaborator

The job x86_64-gnu-llvm-6.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_time:end:079a927d:start=1552133961660549537,finish=1552133964385607011,duration=2725057474
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
$ export GCP_CACHE_BUCKET=rust-lang-ci-cache
Setting environment variables from .travis.yml
---
[01:16:37] .................................................................................................... 3700/5440
[01:16:41] ..................................................ii................................................ 3800/5440
[01:16:43] ....................................................................i............................... 3900/5440
[01:16:46] .................................................................................................... 4000/5440
[01:16:48] ..........................i.F....................................................................... 4100/5440
[01:17:05] .................................................................................................... 4300/5440
[01:17:09] .................................................................................................... 4400/5440
[01:17:12] .................................................................................................... 4500/5440
[01:17:16] .....................................................................................i.............. 4600/5440
---
[01:17:52] 
[01:17:52] ---- [ui] ui/parser/mod_file_not_exist.rs stdout ----
[01:17:52] diff of stderr:
[01:17:52] 
[01:17:52] 1 error[E0583]: file not found for module `not_a_real_file`
[01:17:52] 3    |
[01:17:52] 3    |
[01:17:52] - LL | mod not_a_real_file; //~ ERROR file not found for module `not_a_real_file`
[01:17:52] + LL | mod not_a_real_file;
[01:17:52] 6    |
[01:17:52] 6    |
[01:17:52] 7    = help: name the file either not_a_real_file.rs or not_a_real_file/mod.rs inside the directory "$DIR"
[01:17:52] 
[01:17:52] The actual stderr differed from the expected stderr.
[01:17:52] Actual stderr saved to /checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/parser/mod_file_not_exist/mod_file_not_exist.stderr
[01:17:52] Actual stderr saved to /checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/parser/mod_file_not_exist/mod_file_not_exist.stderr
[01:17:52] To update references, rerun the tests and pass the `--bless` flag
[01:17:52] To only update this specific test, also pass `--test-args parser/mod_file_not_exist.rs`
[01:17:52] error: 1 errors occurred comparing output.
[01:17:52] status: exit code: 1
[01:17:52] status: exit code: 1
[01:17:52] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/parser/mod_file_not_exist.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-C" "prefer-dynamic" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/parser/mod_file_not_exist/a" "-Crpath" "-O" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/parser/mod_file_not_exist/auxiliary" "-A" "unused"
[01:17:52] ------------------------------------------
[01:17:52] 
[01:17:52] ------------------------------------------
[01:17:52] stderr:
[01:17:52] stderr:
[01:17:52] ------------------------------------------
[01:17:52] {"message":"file not found for module `not_a_real_file`","code":{"code":"E0583","explanation":"\nA file wasn't found for an out-of-line module.\n\nErroneous code example:\n\n```ignore (compile_fail not working here; see Issue #43707)\nmod file_that_doesnt_exist; // error: file not found for module\n\nfn main() {}\n```\n\nPlease be sure that a file corresponding to the module exists. If you\nwant to use a module named `file_that_doesnt_exist`, you need to have a file\nnamed `file_that_doesnt_exist.rs` or `file_that_doesnt_exist/mod.rs` in the\nsame directory.\n"},"level":"error","spans":[{"file_name":"/checkout/src/test/ui/parser/mod_file_not_exist.rs","byte_start":23,"byte_end":38,"line_start":3,"line_end":3,"column_start":5,"column_end":20,"is_primary":true,"text":[{"text":"mod not_a_real_file; //~ ERROR file not found for module `not_a_real_file`","highlight_start":5,"highlight_end":20}],"label":null,"suggested_replacement":null,"suggestion_applicability":null,"expansion":null}],"children":[{"message":"name the file either not_a_real_file.rs or not_a_real_file/mod.rs inside the directory \"/checkout/src/test/ui/parser\"","code":null,"level":"help","spans":[],"children":[],"rendered":null}],"rendered":"error[E0583]: file not found for module `not_a_real_file`\n  --> /checkout/src/test/ui/parser/mod_file_not_exist.rs:3:5\n   |\nLL | mod not_a_real_file; //~ ERROR file not found for module `not_a_real_file`\n   |     ^^^^^^^^^^^^^^^\n   |\n   = help: name the file either not_a_real_file.rs or not_a_real_file/mod.rs inside the directory \"/checkout/src/test/ui/parser\"\n\n"}
[01:17:52] {"message":"For more information about this error, try `rustc --explain E0583`.","code":null,"level":"","spans":[],"children":[],"rendered":"For more information about this error, try `rustc --explain E0583`.\n"}
[01:17:52] 
[01:17:52] ------------------------------------------
[01:17:52] 
---
[01:17:52] 
[01:17:52] thread 'main' panicked at 'Some tests failed', src/tools/compiletest/src/main.rs:496:22
[01:17:52] 
[01:17:52] 
[01:17:52] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib" "--run-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib" "--rustc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "--src-base" "/checkout/src/test/ui" "--build-base" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui" "--stage-id" "stage2-x86_64-unknown-linux-gnu" "--mode" "ui" "--target" "x86_64-unknown-linux-gnu" "--host" "x86_64-unknown-linux-gnu" "--llvm-filecheck" "/usr/lib/llvm-6.0/bin/FileCheck" "--host-rustcflags" "-Crpath -O -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--target-rustcflags" "-Crpath -O -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--docck-python" "/usr/bin/python2.7" "--lldb-python" "/usr/bin/python2.7" "--gdb" "/usr/bin/gdb" "--quiet" "--llvm-version" "6.0.0\n" "--system-llvm" "--cc" "" "--cxx" "" "--cflags" "" "--llvm-components" "" "--llvm-cxxflags" "" "--adb-path" "adb" "--adb-test-dir" "/data/tmp/work" "--android-cross-path" "" "--color" "always"
[01:17:52] 
[01:17:52] 
[01:17:52] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
[01:17:52] Build completed unsuccessfully in 0:04:49
[01:17:52] Build completed unsuccessfully in 0:04:49
[01:17:52] make: *** [check] Error 1
[01:17:52] Makefile:48: recipe for target 'check' failed
The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:2419c87a
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
Sat Mar  9 13:37:28 UTC 2019

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@bors
Copy link
Contributor

bors commented Mar 9, 2019

☔ The latest upstream changes (presumably #59050) made this pull request unmergeable. Please resolve the merge conflicts.

@davidtwco
Copy link
Member

I'm happy to r+ this PR, assuming nobody has any issues with it, but I suspect it'll be pretty difficult to land w/out a tree-close given how large it is.

@oli-obk
Copy link
Contributor

oli-obk commented Mar 11, 2019

assuming nobody has any issues with it

I say we do it.

I suspect it'll be pretty difficult to land w/out a tree-close given how large it is.

It should be doable by a quick rebase + bless run while a PR is being processed which doesn't touch stderr files. But a treeclose will make things easier if @petrochenkov has a few contiguous hours to get this working (like if there are random bors failures).

@petrochenkov
Copy link
Contributor Author

Ok, I'll rebase it this evening and try to r+ with high priority.

@petrochenkov petrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Mar 11, 2019
@petrochenkov petrochenkov changed the title [experiment] Filter away test annotations from UI test output Filter away test annotations from UI test output Mar 11, 2019
@petrochenkov
Copy link
Contributor Author

@bors r=davidtwco p=100

@bors
Copy link
Contributor

bors commented Mar 11, 2019

📌 Commit 07f99b9 has been approved by davidtwco

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Mar 11, 2019
@bors
Copy link
Contributor

bors commented Mar 11, 2019

⌛ Testing commit 07f99b9 with merge 7486b9c...

bors added a commit that referenced this pull request Mar 11, 2019
Filter away test annotations from UI test output

If you worked with UI tests for some time you could notice one issue affecting their readability and also readability of diffs when the tests change.
Look at the output of this test.
```rust
fn main() {
    let 1 = 2; //~ ERROR refutable pattern in local binding
}
```
```
error[E0005]: refutable pattern in local binding: `-2147483648i32..=0i32` not covered
 --> src/main.rs:2:9
  |
2 |     let 1 = 2; //~ ERROR refutable pattern in local binding
  |         ^ pattern `-2147483648i32..=0i32` not covered

error: aborting due to previous error

For more information about this error, try `rustc --explain E0005`.
```
You can see that the "refutable pattern in local binding" is duplicated.
One instance is the actual error, and the second instance is the expected error annotation.
This annotation is useful in the test input, but in the output it clutters the text and makes it harder to see what text refers to actual errors and what is just comments, especially if there are many errors in a single test file.

@estebank [reported](#57379 (comment)) using the next trick to avoid the clutter
```rust
fn main() {
    let 1 = 2;
    //~^ ERROR refutable pattern in local binding
}
```
```
error[E0005]: refutable pattern in local binding: `-2147483648i32..=0i32` not covered
 --> src/main.rs:2:9
  |
2 |     let 1 = 2;
  |         ^ pattern `-2147483648i32..=0i32` not covered

error: aborting due to previous error

For more information about this error, try `rustc --explain E0005`.
```
, i.e. using `//~^` and placing the annotation one line below will remove the annotation from the output.

However, this doesn't always works (consider errors with multi-line spans), and shouldn't be necessary in general!
`compiletest` could automatically filter away its own annotations from the output instead.
This is exactly what this PR does.

r? @davidtwco
@bors
Copy link
Contributor

bors commented Mar 12, 2019

☀️ Test successful - checks-travis, status-appveyor
Approved by: davidtwco
Pushing 7486b9c to master...

@bors bors added the merged-by-bors This PR was explicitly merged by bors. label Mar 12, 2019
@bors bors merged commit 07f99b9 into rust-lang:master Mar 12, 2019
@rust-highfive
Copy link
Collaborator

📣 Toolstate changed by #59044!

Tested on commit 7486b9c.
Direct link to PR: #59044

💔 rls on windows: test-pass → test-fail (cc @nrc @Xanewok, @rust-lang/infra).

rust-highfive added a commit to rust-lang-nursery/rust-toolstate that referenced this pull request Mar 12, 2019
Tested on commit rust-lang/rust@7486b9c.
Direct link to PR: <rust-lang/rust#59044>

💔 rls on windows: test-pass → test-fail (cc @nrc @Xanewok, @rust-lang/infra).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
merged-by-bors This PR was explicitly merged by bors. S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants