Set up benchmarks

[totakke]

Sets up a benchmark environment using Libra.

cljam is a performance-sensitive project, so that we have to take care with degradation when changing it. Libra provides an easy way to check the performance, which is similar to clojure.test and lein test.

As a beginning, I added BAM-indexing and depth benchmarks in this PR. lein libra runs all defined benchmarks. If you want to run a specific benchmark, you can use :only selector: lein libra :only cljam.algo.depth-bench.

Libra is still an in-development framework, but I think it has minimum requirements for benchmarking. I plan to add convenient features to it, such as meta-tag selector, CPU/RAM checking, and individual JVMs.

[alumi]

Great feature! 😁

I tried lein libra for 3 times.

It looks fine for create-index-medium-bench.

[1] time: 222.584972 ms, sd: 3.802342 µs
[1] time: 224.401048 ms, sd: 745.758194 ns
[1] time: 223.702554 ms, sd: 14.865770 µs

[2] time: 225.037261 ms, sd: 12.994650 µs
[2] time: 229.320097 ms, sd: 11.242042 µs
[2] time: 226.181123 ms, sd: 12.558158 µs

[4] time: 230.538432 ms, sd: 164.090025 µs
[4] time: 224.096448 ms, sd: 11.534817 µs
[4] time: 221.169244 ms, sd: 10.901597 µs

But results are bit unstable for depth-medium-bench.

[1] time: 7.366831 sec, sd: 174.703666 ms
[1] time: 6.920735 sec, sd: 109.461688 ms
[1] time: 6.803559 sec, sd: 263.615192 ms

[2] time: 7.661819 sec, sd: 357.218477 ms
[2] time: 6.059654 sec, sd: 929.919971 ms
[2] time: 6.113873 sec, sd: 1.531480 sec

[4] time: 3.433568 sec, sd: 288.450404 ms
[4] time: 2.898810 sec, sd: 265.634345 ms
[4] time: 4.247318 sec, sd: 1.011009 sec

Changing dur to c/quick-bench seems to give slightly more stable results,
but it doubled total execution time.

[1] time: 7.175782 sec, sd: 882.365067 ms
[1] time: 7.556264 sec, sd: 1.416275 sec
[1] time: 7.535457 sec, sd: 1.435119 sec

[2] time: 4.431472 sec, sd: 137.034337 ms
[2] time: 4.299521 sec, sd: 202.066247 ms
[2] time: 4.285544 sec, sd: 211.650626 ms

[4] time: 2.791778 sec, sd: 266.641184 ms
[4] time: 2.889218 sec, sd: 77.644608 ms
[4] time: 2.676885 sec, sd: 132.365124 ms

Please confirm if dur is enough for the benchmark.
Thanks.

[totakke]

Thanks. It's a good point.

dur is fast but rough, and c/(quick-)bench is accurate but slow. I'm still worrying which to use. Basically, dur should be selected when accuracy is unnecessary or a benchmark form requires a lot of time. The threshold of time is difficult.

In this case, do you think c/quick-bench is suitable for depth-medium-bench? I'll fix it if so.

[alumi]

It's hard to choose but I think quick-bench is suitable as a default.
It might be a good idea to implement a new lein-libra feature to control accuracy/time at runtime (sometime in future).

[totakke]

I changed dur to c/quick-bench and tweaked the range of depth calculation.

It might be a good idea to implement a new lein-libra feature to control accuracy/time at runtime (sometime in future).

Yes, I think that can be partially realized by meta-tag selector. Automatic change may be more convenient, but it is somewhat difficult. I'll consider it.

[alumi]

Merged. Thanks for the fix!

[totakke]

Thank you for the helpful discussion!