Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Framework for defining which benchmarks are Arrow #4

Open
nealrichardson opened this issue Feb 12, 2021 · 0 comments
Open

Framework for defining which benchmarks are Arrow #4

nealrichardson opened this issue Feb 12, 2021 · 0 comments

Comments

@nealrichardson
Copy link
Contributor

We have two related challenges:

  • Some parameters we want to test are only relevant for Arrow. Some of these, like whether to return an arrow Table or data.frame when reading a file, may be specific to a single benchmark. Others, like environment variables that only affect Arrow, should be controllable at the global level. We don't want to run non-arrow benchmarks parametrized by arrow-only environment variables--that's wasteful/slow.
  • When running benchmarks continuously (on every commit), we only want to run the Arrow benchmarks--no point wasting electrons on other people's (unchanged) code.

What if we added a function (arrow_params()?) similar to default_params that each benchmark can register that indicates which param combinations involve arrow? We can use that in whatever run_all() function we add for conbench, and we can also use it to only add variations for arrow env vars to the combinations that involve arrow.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant