Add wasm_of_ocaml benchmarks with current-bench output

[OlivierNicole]

@vouillon and I (so far, mostly @vouillon) have started to look at ways of improving performance of Wasm code. To keep track of our progress, we need to set up a benchmarking suite that gets run regularly.

This PR makes the repository compatible for monitoring by https://github.com/ocurrent/current-bench/: the benchmark suite is extended with the current-bench output format and I modified the scripts so that Wasm benchmarks are first-class citizens. The existing benchmarking utilities should still work, I ran a few checks to verify that.

This is a first step, we will soon want to add macrobenchmarks as well — presumably by having a benchmarks/sources/macro directory where each subdirectory is a macrobenchmark with its own dune file.

[OlivierNicole]

Note: this is ready for review but I still need to verify details—namely, how current-bench deals with multiple opam files, and what should be the name of the results file—so we should hold off from merging yet.

[OlivierNicole]

Thanks to @punchagan’s advice, this should now hopefully work with current-bench.

[vouillon]

I think make bench needs to build wasm_of_ocaml. The opam file is here only to install external dependencies.

[vouillon]

I would put all the results together and grouping them?

{
  "name": "Wasm_of_ocaml",
  "results": [
    {
      "name": "Microbenchmarks",
      "metrics": [
        {
          "name": "microbenchmark/almabench",
          "value": 1.6087265,
          "units": "s"
        },
        {
          "name": "microbenchmark/bdd",
          "value": 0.31883700000000004,
          "units": "s"
        },

[OlivierNicole]

This will cause all the run times to appear on one unique graph. I’m not sure we want that?

[vouillon]

This will cause all the run times to appear on one unique graph. I’m not sure we want that?

Maybe not, actually, since they have very different running times. This is done for Ocaml, but it is not very readable.

We should still put them in the same list of metrics. This will be more readable when we will add other benchmarks.

Maybe we could report the geometric mean of all the running times as well?

[OlivierNicole]

I think make bench needs to build wasm_of_ocaml. The opam file is here only to install external dependencies.

Actually, I disagree. When you look at the last line of https://github.com/ocurrent/current-bench/blob/b0528d005b34b8ea1ff4cd7f893165ce782ef5db/pipeline/lib/custom_dockerfile.ml#L78-L96, it also installs regular dependencies.

I have rearranged the metrics list as requested.

Maybe we could report the geometric mean of all the running times as well?

A geometric mean of absolute values? Isn’t the geometric mean usually used on relative evolution numbers?

[OlivierNicole]

This repo has been added to the current-bench infra and the results for this PR can be checked here.

[hhugo]

Why do we use an old version of opam ?

[OlivierNicole]

Good question. I took inspiration from https://github.com/ocaml-multicore/picos/blob/main/bench.Dockerfile which does this. I’ll try without this line.

[hhugo]

I don't understand the purpose of wasm_of_ocaml-bench, why don't we install wasm_of_ocaml-compiler instead ?

[OlivierNicole]

Current-bench’s default Dockerfile needed a dedicated opam file for the benchmarks. But now that we have a custom Dockerfile, I can probably drop it.

[hhugo]

I don't understand why we need all the logic in the docker file.
Other open PR seem to fail with

make: *** No rule to make target 'bench'.  Stop.
docker-run failed with exit-code 2

Why isn't it just a matter of added a makefile target ?