Prepare commands that differ, and only run once, depending on the command being timed?

[klundberg]

I have a specific request: We're using hyperfine to measure the difference in compile times between two branches of our codebase. Right now I have two separate hyperfine invocations in a script, and before each of them I check out the specific branch in git that I want to measure compile times for.

I noticed today that hyperfine supports timing and comparing multiple commands in a single invocation and that it provides stats comparing them which is great! But in order for us to use that, we'd need to do a git checkout before each run of the build command, (something like `hyperfine "git checkout branch1; build.sh" "git checkout branch2; build.sh") and I don't want that checkout time to be included in our metrics. Would it be possible to have some sort of global prepare statement per-command that's being run that runs only once, and is different for each command being tested? I imagine that would be a new feature and a nontrivial one at that.

Or would you recommend running a few warmup runs with the checkout in the command to benchmark, or something like that?

Thanks!

[sharkdp]

Thank you for the feedback. That sounds like a useful feature to have.

Did you see the --prepare <cmd> option? This is actually pretty close to what you want, except you can only specify a single command for all benchmark runs:

hyperfine --prepare "git checkout ???" …

We could change that and allow --prepare to be specified either once or exactly N times (where N is the number of benchmarked commands). In the latter case, we would run command-specific cleanup commands. This way, you could use:

hyperfine --prepare "git checkout branch1" "./build.sh" \
          --prepare "git checkout branch2" "./build.sh"

What do you think?

Or would you recommend running a few warmup runs with the checkout in the command to benchmark, or something like that?

If the git checkout branchX time is negligible compared to the build.sh time (in cases where branchX is already checked out!), you can definitely use --warmup:

hyperfine --warmup 1 "git checkout branch1; build.sh" "git checkout branch2; build.sh"

The only problem is that it will perform one unnecessary build for each branch.

[sharkdp]

Another thing that comes really close is the --parameter-scan <VAR> <MIN> <MAX> option.

Imagine for a moment that your branches would actually be called branch1 and branch2. We could then run:

hyperfine \
  --parameter-scan number 1 2 \
  --prepare "git checkout branch{number}" \
  "build.sh"

So another option would be to add a new option that would allow non-numeric parameter runs. Maybe --parameter-list <VAR> <VALUES> which could be used like this:

hyperfine \
  --parameter-list branch_name master,feature1,bugfix,my-test-feature \
  --prepare "git checkout {branch_name}" \
  "build.sh"

[piyushrungta25]

I will add another way we can address a lot of issues regarding parameterized commands by splitting the benchmarking and reporting in two separate commands.

For eg. you can invoke hyperfine with different arguments multiple time and the results will be appended in a session file

hyperfine --session bench.json --prepare "cmd" "./build.sh"
hyperfine --session bench.json --prepare "cmd2" "./build.sh"

and then a report sub-command can print the pretty comparison stats and everything from that file.

hyperfine report --session bench.json

This way we can loop over parameters in bash easily. If --session is not specified, then it can continue to work as it does today.

[klundberg]

These all sound like great solutions. Today our script generates two json files with something along the lines of what @piyushrungta25 showed in that example, and I'm manually doing statistical calculations to get the difference in the means/std deviations, so some way to report on either a session file, or on two or more independent exports would be nice (and that seems valuable for comparing results over longer time periods, independent of my initial request, which is something we're also currently planning to do).

However It'd be amazing to have one of the other options, either --parameter-list or multiple --prepare statements. --parameter-list feels like it would be a simpler thing to implement to me (only based on intuition, as I don't know Rust and haven't read the code). I suspect one could use that to mimic something along the lines of multiple prepare/cleanup statements if the prepare statements run scripts that accept the parameter list as an argument or environment variable, and branch based on that. In fact, I could probably use --parameter-scan to do that right now, but --parameter-list would definitely make that much easier to write and maintain.

We have some git hooks right now that run on every checkout, and those take some time to run, so --warmup may not be the right option for us (even though we currently do use it to make sure any file system caching doesn't impact our build).

Thank you for the responses and reception to my request! We have our workarounds for now, but if we ever do this analysis in the future, one or more of these options would definitely be welcome!

[iamsauravsharma]

I would like to take this issue if no one is currently working on this. I think this issue have 2 feature request:-

1. Allow multiple prepare option.
2. Run prepare command only once

I would like to implement out feature 1 by supporting multiple prepare option and second feature by addition of new option maybe --once/-o to prepare for only one time.

[sharkdp]

I will add another way we can address a lot of issues regarding parameterized commands by splitting the benchmarking and reporting in two separate commands.

For eg. you can invoke hyperfine with different arguments multiple time and the results will be appended in a session file

I'd rather not follow that path. It would definitely be very powerful, but I would really like hyperfine to be a simple, single-invocation command that is easy to use.

[sharkdp]

This is now supported via #218 by @iamsauravsharma. However, the git checkout command would be run before each build (but not included in the benchmark, of course). see also #219

[sharkdp]

Released in v1.8.0.

[klundberg]

Wonderful, thank you so much for implementing this!

…

-- Kevin Lundberg

On Oct 13, 2019, at 10:59 AM, David Peter ***@***.***> wrote:  Released in v1.8.0. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.