Add a function to generate nice reports

[jebej]

I have started using this package to test a module I'm making and it works really well!

I was thinking that it would be useful to have function to generate a report from the results of a suite. What I ended up doing was to go look at the Nanosoldier repo and copy most of the code there (the printreport function). Having such a function in this repository would be useful.

[jrevels]

That's probably a good idea. I think I'd want to alter the current approach to spit out HTML rather than markdown, leaving open the possibility for cool interactive JS stuff a la criterion's awesome reports.

[jrevels]

cc @ivirshup, you're probably working on other things but just in case you have ideas

[ahwillia]

+1 for this. I'd be happy to contribute a little bit here.

In an ideal world, I would love the following to work:

versions = ["v0.0.1", "3743c273f4439833787522242efdcda87951f6d1", "v0.0.2"]
results = Pkg.benchmark("Foo",versions)

Edit: I've opened a new issue to discuss this in detail (and deleted much of this comment to reduce noise): #16

[jrevels]

@ahwillia This issue is just for generating reports from benchmark data, which is separate from Base.Pkg support for BenchmarkTools (though both would be nice). From previous discussions on that front, it probably doesn't make sense right now for Base.Pkg to depend on BenchmarkTools. That might change in the (far) future if Base moves to a more decentralized structure.

BenchmarkTools could natively support the functionality you mentioned, instead of it being implemented in Base.Pkg. Feel free to open an issue for it.

[ahwillia]

BenchmarkTools could natively support the functionality you mentioned

Yes, this is what I was hoping for in the short term.

Feel free to open an issue for it.

Done. Apologies for commenting here first. I misinterpreted the scope of this issue too broadly.

[mforets]

Perhaps not too difficult would be to generate a table similar to that in export_markdown but with the signature export_markdown(::IO, ::BenchmarkGroup). Or this functionality exists already? I'm not familiar with the internals of BenchmarkTools.