Add wallclock benchmarks

[overlookmotel]

Problem

CodSpeed is good, but has some anomalies.

In particular:

1. All system calls (e.g. system allocator) are "free" on Codspeed's measures.
2. They say it doesn't take into account the branch predictor.

Mispredicted branches can be a significant perf hit, which we're failing to measure. In particular, I have some ideas to replace branching in lexer with straight-line code (#108). I suspect this could be a significant gain, but it won't register on current CodSpeed benchmarks - so we can't evaluate this at present.

Possible solution

Introduce wallclock benchmarks (not run with Valgrind) in addition to the existing benchmarks.

How?

Can use the same hack I wrote to run NAPI benchmarks as normal wallclock benchmarks and get the results into CodSpeed.

An improvement would be if it's possible to synthesize fake .out files to send to CodSpeed.

[Boshen]

I also want things like cpu cycles, max rss recorded, binary output ... everything we care about.

[overlookmotel]

I also want things like cpu cycles, max rss recorded, binary output ... everything we care about.

Yes! Please see #6.

[overlookmotel]

Boshen pointed out that problem with wallclock benchmarks on CI is that you get a different machine, potentially with a different CPU etc on each benchmark run. So that introduces variance.

Rolldown is working around that by running benchmarks twice each time - once for current commit/PR, and once for base - and then comparing the two. Both run in series on same machine, so that removes the source of variance.

Notes:

1. Problem is you only get a relative measure each time (new vs old on this machine). So to get this into CodSpeed, would need to fetch benchmark measure that was submitted to CodSpeed, and then adjust up/down by the relative result for run. i.e. new_uploaded_time = previously_uploaded_time * time_for_new_just_measured / time_for_old_just_measured.
2. Need to build each benchmark twice. But could build both in parallel in separate jobs, and then run them in series in a single job. Or building 2 in parallel on same machine may actually be OK, because the last ~50% of build time for benchmarks is on a single thread anyway.

[Boshen]

Continuous wall clock benchmark isn't feasible, we had this setup before codspeed. But, we can add a conditional ci job trigger to turn off codspeed and measure against main branch.

[overlookmotel]

Roughly how much variance were you seeing in e.g. parser benchmarks prior to CodSpeed?

[overlookmotel]

Background on why CodSpeed doesn't take into account branch predictor:

CodSpeed uses Valgrind, which doesn't simulate branch prediction by default: https://valgrind.org/docs/manual/cg-manual.html#cg-manual.cache-branch-sim

https://discord.com/channels/1065233827569598464/1314623192257790045

[camchenry]

I also want things like cpu cycles, max rss recorded, binary output ... everything we care about.

It looks like the iai-callgrind library could deliver this for us. It would require us writing new benchmarks though, it doesn't seem like we could reuse our existing Criterion benchmarks. Appears to be very similar to what CodSpeed does for us, but gives us the control of outputting whatever we want:

• Precise: High-precision measurements of Instruction counts and many other metrics allow you to reliably detect very small optimizations and regressions of your code.
• Consistent: Iai-Callgrind can take accurate measurements even in virtualized CI environments and make them comparable between different systems completely negating the noise of the environment.
• Fast: Each benchmark is only run once, which is usually much faster than benchmarks which measure execution and wall-clock time. Benchmarks measuring the wall-clock time have to be run many times to increase their accuracy, detect outliers, filter out noise, etc.

It doesn't handle wall-clock times at all (like CodSpeed):

If you need wall-clock times, Iai-Callgrind cannot help you much. The estimation of cpu cycles merely correlates to wall-clock times but is not a replacement for wall-clock times. The cycles estimation is primarily designed to be a relative metric to be used for comparison.

However, it gives us many other kinds of events which would be useful to track for at least a subset of code (e.g., parser and lexer):

• instruction counts
• estimated CPU cycles
• data cache reads/writes
• branch executions, mispredictions
• L1/LL cache hits
• quantifying temporal/spatial locality for L1/LL caches

[overlookmotel]

Would this be any more accurate on branch predictor statistics than CodSpeed is? (see #5 (comment)).

From what I read, you can enable branch predictor emulation in Valgrind, but the emulation is based on a really old processor, like Pentium 4 or something. CPUs have come a long way since them, so it may not be a very accurate reflection of what happens on a modern CPU.

If we want to add more benchmarking "angles", I would tend to think we should avoid emulators - CodSpeed already has that covered for us - and try to get measures off a real CPU.

[camchenry]

Would this be any more accurate on branch predictor statistics than CodSpeed is? (see #5 (comment)).

I'm not sure, I haven't seen CodSpeed expose any metrics other than time, but I don't think iai-callgrind would be much different than what CodSpeed does, since it's basically the same approach and even using the same tools.

From what I read, you can enable branch predictor emulation in Valgrind, but the emulation is based on a really old processor, like Pentium 4 or something. CPUs have come a long way since them, so it may not be a very accurate reflection of what happens on a modern CPU.

Looks like Cachegrind is mostly the same as Callgrind in this regard:

Callgrind's cache simulation is based on that of Cachegrind
-- https://valgrind.org/docs/manual/cl-manual.html#cl-manual.functionality

So yeah, I don't think it's going to be an improvement in this area at all.

If we want to add more benchmarking "angles", I would tend to think we should avoid emulators - CodSpeed already has that covered for us - and try to get measures off a real CPU.

Yes, I feel like we've already pushed the limits of CodSpeed (and the Valgrind/Callgrind/Cachegrind approach in general). There is the CodSpeed walltime benchmarking, which I've tried to utilize previously: oxc-project/oxc#9934, but was unsuccessful as it has somewhat limited usage available for free, and the criterion fork (https://github.com/Boshen/criterion2.rs) we're using isn't supported easily it sounds like.

So that's why I am trialing just running some simple Criterion benchmarks in CI (oxc-project/oxc#11874) to give us at least some real-world performance numbers even if it's somewhat noisy. Ultimately, I feel like I get the best results from just running the benchmarks on my laptop. Maybe we should invest in more tooling to make that a simple ./run-the-benchmarks-and-compare script.