Graph: Combined Jitter Scatter-plot and Box-plot

[redknightlois]

With all the data that any benchmark gets as results and the amount of times operations tend to be executed, its not a bad idea to also allow outputting this kind of graph

The explanation of the graph is here: https://www.pvk.ca/Blog/2012/07/03/binary-search-star-eliminates-star-branch-mispredictions/

[dbgoytia]

Hey @AndreyAkinshin ! I was wondering if this issue is still relevant? I'd like to contribute, although I might need a bit of help getting started

[AndreyAkinshin]

@dbgoytia yes, it's still relevant.
At the moment, BenchmarkDotNet doesn't have its own plotting subsystem. The only way to generate plots is [RPlotExporter]. It uses locally installed R and a custom R script that could be found here. Thus, the only way to support a new kind of plots is to add the corresponding changes in the R script (no C#, sorry).

P.S. It would be nice to have own plotting system so that we can draw charts without installed R. However, we don't want to add any additional dependencies to other NuGet packages. Thus, we should implement such a plotting logic from scratch. It's a huge tasks that require a lot of efforts, so we don't have it in the roadmap for the nearest future. Also, it makes sense to implement it on the perfolizer side rather than in BenchmarkDotNet.

[FlatlinerDOA]

Can I suggest a separate Nuget package is created to support it BenchmarkDotNet.Exporters.Plot or something, and use a library like ScottPlot at least for basic box plots - https://scottplot.net/cookbook/5.0/

[AndreyAkinshin]

@FlatlinerDOA sounds good to me. While I'm still against adding NuGet dependencies to the main package without a good reason, we have no limitations on creating new packages with any dependencies.

[FlatlinerDOA]

@AndreyAkinshin I've got a basic Bar Plot outputting for now in a separate package called BenchmarkDotNet.Exporters.Plotting. Is your preference to get all plots supported first (in one large PR). Or just get a preview package set up and add plot types one by one?

[AndreyAkinshin]

@FlatlinerDOA thanks again for the PR and sorry for such a long delay with merging.
It's better to add plots one-by-one. I will include the new NuGet package in the upcoming release.

[FlatlinerDOA]

@AndreyAkinshin no worries, I have created two PR's: One #2611 is a small to fix a bug in the result filtering, and #2614 for adding Box Plots.