test: Display benchmark results with thousands separators #26068

bluss · 2015-06-07T17:27:09Z

test: Display benchmark results with thousands separators

Example display:

running 9 tests
test a ... bench:           0 ns/iter (+/- 0)
test b ... bench:          52 ns/iter (+/- 0)
test c ... bench:          88 ns/iter (+/- 0)
test d ... bench:         618 ns/iter (+/- 111)
test e ... bench:       5,933 ns/iter (+/- 87)
test f ... bench:      59,280 ns/iter (+/- 1,052)
test g ... bench:     588,672 ns/iter (+/- 3,381)
test h ... bench:   5,894,227 ns/iter (+/- 303,489)
test i ... bench:  59,112,382 ns/iter (+/- 1,500,110)

Fixes #10953
Fixes #26109

rust-highfive · 2015-06-07T17:27:23Z

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @brson (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. The way Github handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

bluss · 2015-06-07T17:27:53Z

r? @huonw

I think scientific notation is too hard to read. The nicest improvement is rounding to sig figs, units are a bit tricky to notice too.

bluss · 2015-06-08T17:43:18Z

Maybe four sigfigs & 1 sigfig deviation is better

example

running 9 tests
test a ... bench:         0 ns/iter (+/- 0)
test b ... bench:        52 ns/iter (+/- 0)
test c ... bench:        87 ns/iter (+/- 0)
test d ... bench:       617 ns/iter (+/- 1)
test e ... bench:     5.913 us/iter (+/- 0.02)
test f ... bench:     58.87 us/iter (+/- 0.03)
test g ... bench:     588.7 us/iter (+/- 0.7)
test h ... bench:     5.885 ms/iter (+/- 0.03)
test i ... bench:     58.88 ms/iter (+/- 0.2)

Example from real benchmarks

running 10 tests
test slice_iter              ... bench:        24 ns/iter (+/- 2)
test slice_iter_rev          ... bench:        24 ns/iter (+/- 0)
test stride_iter             ... bench:        24 ns/iter (+/- 1)
test stride_iter_rev         ... bench:        24 ns/iter (+/- 0)
test zip_loop                ... bench:       923 ns/iter (+/- 53)
test zip_loop3               ... bench:     1.372 us/iter (+/- 0.1)
test zip_slices_default_zip  ... bench:       935 ns/iter (+/- 9)
test zip_slices_default_zip3 ... bench:     1.383 us/iter (+/- 0.08)
test zip_slices_ziptuple     ... bench:       941 ns/iter (+/- 40)
test zipslices               ... bench:       942 ns/iter (+/- 167)

test result: ok. 0 passed; 0 failed; 0 ignored; 10 measured

huonw · 2015-06-08T22:37:48Z

Hm, as you say, the distinction between units is very subtle, especially the two most common unfortunately (n vs. u).

Gankra · 2015-06-08T22:43:35Z

More than anything it seems harder to "follow the trend" visually. Actual
charts and diagrams seems like a richer way to handle this sort of thing.

On Mon, Jun 8, 2015 at 3:37 PM, Huon Wilson notifications@github.com
wrote:

Hm, as you say, the distinction between units is very subtle, especially
the two most common unfortunately (n vs. u).

—
Reply to this email directly or view it on GitHub
#26068 (comment).

bluss · 2015-06-08T23:00:33Z

Let's do thousands separators instead?

huonw · 2015-06-08T23:09:04Z

That might be better yeah.

bluss · 2015-06-08T23:23:55Z

They look like this

running 9 tests
test a ... bench:           0 ns/iter (+/- 0)
test b ... bench:          52 ns/iter (+/- 0)
test c ... bench:          88 ns/iter (+/- 0)
test d ... bench:         618 ns/iter (+/- 111)
test e ... bench:       5,933 ns/iter (+/- 87)
test f ... bench:      59,280 ns/iter (+/- 1,052)
test g ... bench:     588,672 ns/iter (+/- 3,381)
test h ... bench:   5,894,227 ns/iter (+/- 303,489)
test i ... bench:  59,112,382 ns/iter (+/- 1,500,110)

test result: ok. 0 passed; 0 failed; 0 ignored; 9 measured

I think that's good enough. We could round it, but more correct was already harder to read.

bluss · 2015-06-08T23:30:20Z

Updated branch. I guess the thousands formatting was pretty quick and dirty.

huonw · 2015-06-09T00:17:50Z

src/libtest/lib.rs

+        let base = 10_usize.pow(pow);
+        if pow == 0 || n / base != 0 {
+            if first {
+                output.write_fmt(format_args!("{}", n / base)).ok();


ok -> unwrap()?

I guess that's more idiomatic. On a string it doesn't matter.

Example display: ``` running 9 tests test a ... bench: 0 ns/iter (+/- 0) test b ... bench: 52 ns/iter (+/- 0) test c ... bench: 88 ns/iter (+/- 0) test d ... bench: 618 ns/iter (+/- 111) test e ... bench: 5,933 ns/iter (+/- 87) test f ... bench: 59,280 ns/iter (+/- 1,052) test g ... bench: 588,672 ns/iter (+/- 3,381) test h ... bench: 5,894,227 ns/iter (+/- 303,489) test i ... bench: 59,112,382 ns/iter (+/- 1,500,110) ``` Fixes rust-lang#10953 Fixes rust-lang#26109

bluss · 2015-06-09T00:51:06Z

Updated to use .unwrap().

huonw · 2015-06-09T01:14:13Z

@bors r+

(Now there's just the question of localising it properly...)

bors · 2015-06-09T01:14:14Z

📌 Commit 2b50c15 has been approved by huonw

bors · 2015-06-09T03:20:31Z

⌛ Testing commit 2b50c15 with merge f1313da...

bors · 2015-06-09T04:28:56Z

💔 Test failed - auto-mac-64-opt

alexcrichton · 2015-06-09T06:09:43Z

@bors: retry

On Mon, Jun 8, 2015 at 9:28 PM, bors notifications@github.com wrote:

[image: 💔] Test failed - auto-mac-64-opt
http://buildbot.rust-lang.org/builders/auto-mac-64-opt/builds/5266

—
Reply to this email directly or view it on GitHub
#26068 (comment).

bors · 2015-06-09T09:12:10Z

⌛ Testing commit 2b50c15 with merge a9f50bd...

test: Display benchmark results with thousands separators Example display: ``` running 9 tests test a ... bench: 0 ns/iter (+/- 0) test b ... bench: 52 ns/iter (+/- 0) test c ... bench: 88 ns/iter (+/- 0) test d ... bench: 618 ns/iter (+/- 111) test e ... bench: 5,933 ns/iter (+/- 87) test f ... bench: 59,280 ns/iter (+/- 1,052) test g ... bench: 588,672 ns/iter (+/- 3,381) test h ... bench: 5,894,227 ns/iter (+/- 303,489) test i ... bench: 59,112,382 ns/iter (+/- 1,500,110) ``` Fixes #10953 Fixes #26109

bors · 2015-06-09T10:55:03Z

☀️ Test successful - auto-linux-32-nopt-t, auto-linux-32-opt, auto-linux-64-nopt-t, auto-linux-64-opt, auto-linux-64-x-android-t, auto-mac-32-opt, auto-mac-64-nopt-t, auto-mac-64-opt, auto-win-gnu-32-nopt-t, auto-win-gnu-32-opt, auto-win-gnu-64-nopt-t, auto-win-gnu-64-opt

rust-highfive assigned brson Jun 7, 2015

rust-highfive assigned huonw and unassigned brson Jun 7, 2015

bluss force-pushed the bench-sigfigs branch from 315f2c6 to c2c4ef0 Compare June 7, 2015 17:31

bluss mentioned this pull request Jun 8, 2015

bench should put deliminators on nanosecond numbers #26109

Closed

bluss force-pushed the bench-sigfigs branch from c2c4ef0 to cd4650f Compare June 8, 2015 23:29

bluss changed the title ~~test: Display benchmark results rounded to significant figures~~ test: Display benchmark results with thousands separators Jun 8, 2015

huonw reviewed Jun 9, 2015
View reviewed changes

bluss force-pushed the bench-sigfigs branch from cd4650f to 2b50c15 Compare June 9, 2015 00:50

bors merged commit 2b50c15 into rust-lang:master Jun 9, 2015

bluss deleted the bench-sigfigs branch June 9, 2015 10:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: Display benchmark results with thousands separators #26068

test: Display benchmark results with thousands separators #26068

bluss commented Jun 7, 2015

rust-highfive commented Jun 7, 2015

bluss commented Jun 7, 2015

bluss commented Jun 8, 2015

huonw commented Jun 8, 2015

Gankra commented Jun 8, 2015

bluss commented Jun 8, 2015

huonw commented Jun 8, 2015

bluss commented Jun 8, 2015

bluss commented Jun 8, 2015

huonw Jun 9, 2015

bluss Jun 9, 2015

bluss commented Jun 9, 2015

huonw commented Jun 9, 2015

bors commented Jun 9, 2015

bors commented Jun 9, 2015

bors commented Jun 9, 2015

alexcrichton commented Jun 9, 2015

bors commented Jun 9, 2015

bors commented Jun 9, 2015

test: Display benchmark results with thousands separators #26068

test: Display benchmark results with thousands separators #26068

Conversation

bluss commented Jun 7, 2015

rust-highfive commented Jun 7, 2015

bluss commented Jun 7, 2015

bluss commented Jun 8, 2015

huonw commented Jun 8, 2015

Gankra commented Jun 8, 2015

bluss commented Jun 8, 2015

huonw commented Jun 8, 2015

bluss commented Jun 8, 2015

bluss commented Jun 8, 2015

huonw Jun 9, 2015

Choose a reason for hiding this comment

bluss Jun 9, 2015

Choose a reason for hiding this comment

bluss commented Jun 9, 2015

huonw commented Jun 9, 2015

bors commented Jun 9, 2015

bors commented Jun 9, 2015

bors commented Jun 9, 2015

alexcrichton commented Jun 9, 2015

bors commented Jun 9, 2015

bors commented Jun 9, 2015