add UI testing framework #33588

nikomatsakis · 2016-05-12T14:04:04Z

This adds a framework for capturing and tracking the precise output of rustc, which allows us to check all manner of minor details with the output. It's pretty strict right now -- the output must match almost exactly -- and hence maybe a bit too strict. But I figure we can add wildcards or whatever later. There is also a script intended to make updating the references easy, though the script could make things a bit easier (in particular, it'd be nice if it would find the build directory for you automatically).

One thing I was wondering about is the best way to test colors. Since windows doesn't embed those in the output stream, this test framework can't test colors on windows -- so I figure we can just write tests that are ignored on windows and which pass --color=always or whatever to rustc.

cc @jonathandturner
r? @alexcrichton

alexcrichton · 2016-05-12T16:32:24Z

src/tools/compiletest/src/runtest.rs

+    fn compare_output(&self, kind: &str, actual: &[u8], expected: &[u8]) -> usize {
+        if self.config.verbose {
+            println!("normalized {}:\n{}\n", kind, str::from_utf8(actual).unwrap_or("not utf8"));
+            println!("expected {}:\n{}\n", kind, str::from_utf8(expected).unwrap_or("not utf8"));


May be worth using String::from_utf8_lossy instead of "not uf8" on invalid utf-8

retep998 · 2016-05-12T16:47:09Z

It is possible to create a console screen buffer, redirect rustc to that, and then read the output including all the color attributes. A bit complicated, but nonetheless very much possible.

alexcrichton · 2016-05-12T16:48:53Z

Nice! Some thoughts:

Could you add this to rustbuild as well? If you search for "rpass" it should just be a similar set of modifications to that.
Cargo has a similar method for diffing output, some other normalizations it applies are removing all carriage returns and making tabs more readable as <tab> to match in the output. Neither of those may apply too much here.
Cargo's method calculates differences between lines of input and output to print out out only the lines that mismatch. The only clever thing it does here is to use [..] as a placeholder for "any text"

I'd personally want to get rustbuild running these tests as well as printing the actual/expected on failure, but other than that the remaining pieces seem like they're nice-to-have and we can add later. Having a script to blanket overwrite all the previous output may also be a bit heavy-handed? In theory these shouldn't be too hard to run by hand, right? I'd just be worried that having these scripts checked in makes it "too easy" to blanket change the rustc output without thinking much.

nikomatsakis · 2016-05-13T00:52:42Z

@alexcrichton

Cargo's method calculates differences between lines of input and output to print out out only the lines that mismatch. The only clever thing it does here is to use [..] as a placeholder for "any text"

I may try to at least get diffs of the output working.

I'd personally want to get rustbuild running these tests as well as printing the actual/expected on failure, but other than that the remaining pieces seem like they're nice-to-have and we can add later.

I agree those are necessary.

Having a script to blanket overwrite all the previous output may also be a bit heavy-handed?...I'd just be worried that having these scripts checked in makes it "too easy" to blanket change the rustc output without thinking much.

This was sort of the goal =) I figured we'd make it easy to update, then use review + git diff to check the results.

@retep998

It is possible to create a console screen buffer, redirect rustc to that, and then read the output including all the color attributes. A bit complicated, but nonetheless very much possible.

Cool. Nonetheless, after discussing with @jonathandturner and @alexcrichton, we were thinking we'd just move all color checking to operate at a more semantic level rather than using this framework (i.e., if we get more "presentation" info in the JSON as discussed here, we can check that this is as expected, and then we'll just trust that the translation to ANSI/Windows goes swimmingly).

alexcrichton · 2016-05-13T06:08:13Z

@nikomatsakis all sounds good to me!

nikomatsakis · 2016-05-13T14:30:05Z

@alexcrichton ok, I did the following:

normalize \r\n to \n (I have tested this on windows and this normalization wasn't needed, but it seems like a good idea)
normalize tabs to \r
ensured that we always dump the (normalized) actual and expected output
print a simple diff using cargo's code (maybe worth pulling in code from one of these projects instead, but that can happen later)
added to rust-build (but that...is not quite working yet, still investigating)

I chose not to:

keep the [..] wildcard:
- I hope we can get away with normalization: otherwise, the compiler can't automatically produce the reference for you to copy in.
- but it'd be simple enough to add back. maybe I should just add it proactively.

alexcrichton · 2016-05-13T17:14:39Z

src/bootstrap/build/step.rs

@@ -111,6 +111,8 @@ macro_rules! targets {
            (check_pfail, CheckPFail { compiler: Compiler<'a> }),
            (check_codegen, CheckCodegen { compiler: Compiler<'a> }),
            (check_codegen_units, CheckCodegenUnits { compiler: Compiler<'a> }),
+            (check_incremental, CheckIncremental { compiler: Compiler<'a> }),
+            (check_ui, CheckUi { compiler: Compiler<'a> }),


Could you be sure to add these to the blanket Check target below?

alexcrichton · 2016-05-13T17:16:26Z

Looks good to me! Just one minor nit and otherwise r=me

nikomatsakis · 2016-05-13T22:24:11Z

@bors r=alexcrichton

bors · 2016-05-13T22:24:12Z

📌 Commit 3ff521b has been approved by alexcrichton

bors · 2016-05-14T21:51:38Z

⌛ Testing commit 3ff521b with merge 9c87f71...

bors · 2016-05-14T21:59:01Z

⛄ The build was interrupted to prioritize another pull request.

bors · 2016-05-14T22:32:47Z

⌛ Testing commit 3ff521b with merge 9bbfe4d...

bors · 2016-05-14T23:35:52Z

💔 Test failed - auto-linux-64-opt-rustbuild

sanxiyn · 2016-05-15T06:21:47Z

This failed tidy.

Rollup of 27 pull requests - Successful merges: #33342, #33393, #33415, #33475, #33517, #33533, #33534, #33565, #33580, #33584, #33585, #33588, #33590, #33591, #33593, #33598, #33600, #33602, #33603, #33604, #33605, #33607, #33612, #33620, #33633, #33634, #33635 - Failed merges: #33578

nikomatsakis · 2016-05-16T16:51:00Z

@bors r=alexcrichton

bors · 2016-05-16T16:51:01Z

📌 Commit 474eb38 has been approved by alexcrichton

nikomatsakis · 2016-05-16T20:49:59Z

@bors r=acrichto

bors · 2016-05-16T20:50:00Z

📌 Commit 24cfa1e has been approved by acrichto

bors · 2016-05-16T21:41:51Z

⌛ Testing commit 24cfa1e with merge cd6a400...

@alexcrichton

add UI testing framework This adds a framework for capturing and tracking the precise output of rustc, which allows us to check all manner of minor details with the output. It's pretty strict right now -- the output must match almost exactly -- and hence maybe a bit too strict. But I figure we can add wildcards or whatever later. There is also a script intended to make updating the references easy, though the script could make things a *bit* easier (in particular, it'd be nice if it would find the build directory for you automatically). One thing I was wondering about is the best way to test colors. Since windows doesn't embed those in the output stream, this test framework can't test colors on windows -- so I figure we can just write tests that are ignored on windows and which pass `--color=always` or whatever to rustc. cc @jonathandturner r? @alexcrichton

bors · 2016-05-17T01:39:57Z

☀️ Test successful - auto-linux-32-nopt-t, auto-linux-32-opt, auto-linux-32cross-opt, auto-linux-64-cargotest, auto-linux-64-cross-armhf, auto-linux-64-cross-armsf, auto-linux-64-cross-freebsd, auto-linux-64-cross-netbsd, auto-linux-64-debug-opt, auto-linux-64-nopt-t, auto-linux-64-opt, auto-linux-64-opt-mir, auto-linux-64-opt-rustbuild, auto-linux-64-x-android-t, auto-linux-cross-opt, auto-linux-musl-64-opt, auto-mac-32-opt, auto-mac-64-nopt-t, auto-mac-64-opt, auto-mac-64-opt-rustbuild, auto-mac-cross-ios-opt, auto-win-gnu-32-nopt-t, auto-win-gnu-32-opt, auto-win-gnu-32-opt-rustbuild, auto-win-gnu-64-nopt-t, auto-win-gnu-64-opt, auto-win-msvc-32-cross-opt, auto-win-msvc-32-opt, auto-win-msvc-64-cargotest, auto-win-msvc-64-opt, auto-win-msvc-64-opt-mir, auto-win-msvc-64-opt-rustbuild

rust-highfive assigned alexcrichton May 12, 2016

nikomatsakis force-pushed the compiletest-ui branch from 6499374 to cc19830 Compare May 12, 2016 14:05

alexcrichton reviewed May 12, 2016
View reviewed changes

alexcrichton reviewed May 13, 2016
View reviewed changes

nikomatsakis added 4 commits May 13, 2016 15:22

add UI testing framework

cb112dc

dump outputs, diff on UI test failure

225fa0f

modify rust-build to support incremental, ui tests

1ba4e7b

check check-ui and check-incremental in check

3ff521b

nikomatsakis force-pushed the compiletest-ui branch from 5f73f52 to 3ff521b Compare May 13, 2016 22:23

eddyb mentioned this pull request May 14, 2016

Rollup of 27 pull requests #33643

Merged

pacify the mercilous tidy

24cfa1e

nikomatsakis force-pushed the compiletest-ui branch from 474eb38 to 24cfa1e Compare May 16, 2016 20:48

bors merged commit 24cfa1e into rust-lang:master May 17, 2016

sanxiyn mentioned this pull request Aug 16, 2016

Devise a way to test spans #12182

Closed

nikomatsakis deleted the compiletest-ui branch October 3, 2016 14:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add UI testing framework #33588

add UI testing framework #33588

nikomatsakis commented May 12, 2016

alexcrichton May 12, 2016

retep998 commented May 12, 2016

alexcrichton commented May 12, 2016

nikomatsakis commented May 13, 2016

alexcrichton commented May 13, 2016

nikomatsakis commented May 13, 2016

alexcrichton May 13, 2016

alexcrichton commented May 13, 2016

nikomatsakis commented May 13, 2016

bors commented May 13, 2016

bors commented May 14, 2016

bors commented May 14, 2016

bors commented May 14, 2016

bors commented May 14, 2016

sanxiyn commented May 15, 2016

nikomatsakis commented May 16, 2016

bors commented May 16, 2016

nikomatsakis commented May 16, 2016

bors commented May 16, 2016

bors commented May 16, 2016

bors commented May 17, 2016

add UI testing framework #33588

add UI testing framework #33588

Conversation

nikomatsakis commented May 12, 2016

alexcrichton May 12, 2016

Choose a reason for hiding this comment

retep998 commented May 12, 2016

alexcrichton commented May 12, 2016

nikomatsakis commented May 13, 2016

alexcrichton commented May 13, 2016

nikomatsakis commented May 13, 2016

alexcrichton May 13, 2016

Choose a reason for hiding this comment

alexcrichton commented May 13, 2016

nikomatsakis commented May 13, 2016

bors commented May 13, 2016

bors commented May 14, 2016

bors commented May 14, 2016

bors commented May 14, 2016

bors commented May 14, 2016

sanxiyn commented May 15, 2016

nikomatsakis commented May 16, 2016

bors commented May 16, 2016

nikomatsakis commented May 16, 2016

bors commented May 16, 2016

bors commented May 16, 2016

bors commented May 17, 2016