A report that compares runs

[marcrasi]

It would be useful to get a report of the relative difference between two runs.

Some existing tools:

• google/benchmark's compare.py. (I tried running this on the json from swift-benchmark but it didn't print anything. I guess the format is a bit different.)
• The Swift compiler's benchmark report. Example.

[shabalind]

I've had at a look at google/benchmark's compare.py and it looks like it's not flexible enough to support our user-defined columns.

So we should do our own comparison script:

• It takes a list of json or csv files that represent individual runs. First one is used as a baseline.
• It takes an optional list of columns to compare --columns column1,column2,....
• It takes an optional --format $format where format is console, json, csv.

The console output will look like:

$ compare before.json after.json --columns time

name   column  before  after   relative
---------------------------------------
bench1 time    100 ns  200 ns  2.0x

It would probably nice to do it in Swift as well, because we can reuse table and format reporting.