feat: Add support for --mem-pool-type and --memory-limit options to multiple benchmarks #14642
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Kontinuation
changed the title
Kontinuation
changed the title
Kontinuation
changed the title
|
|
alamb
approved these changes
Feb 13, 2025
2010YOUY01
approved these changes
Feb 14, 2025
|
Thanks again @Kontinuation and @2010YOUY01 |
|
This PR is merged but for some reason the github ui is not showing it: |
jonahgao
pushed a commit
to jonahgao/datafusion
that referenced
this pull request
Feb 14, 2025
…ultiple benchmarks (apache#14642) * Add support --mem-pool-type and --memory-limit options for all benchmarks * Add --sort-spill-reservation-bytes option
acking-you
pushed a commit
to acking-you/arrow-datafusion
that referenced
this pull request
Mar 27, 2025
…ultiple benchmarks (apache#14642) * Add support --mem-pool-type and --memory-limit options for all benchmarks * Add --sort-spill-reservation-bytes option
acking-you
pushed a commit
to acking-you/arrow-datafusion
that referenced
this pull request
Mar 28, 2025
…ultiple benchmarks (apache#14642) * Add support --mem-pool-type and --memory-limit options for all benchmarks * Add --sort-spill-reservation-bytes option
acking-you
pushed a commit
to acking-you/arrow-datafusion
that referenced
this pull request
Mar 29, 2025
…ultiple benchmarks (apache#14642) * Add support --mem-pool-type and --memory-limit options for all benchmarks * Add --sort-spill-reservation-bytes option
acking-you
pushed a commit
to acking-you/arrow-datafusion
that referenced
this pull request
Mar 29, 2025
…ultiple benchmarks (apache#14642) * Add support --mem-pool-type and --memory-limit options for all benchmarks * Add --sort-spill-reservation-bytes option
acking-you
pushed a commit
to acking-you/arrow-datafusion
that referenced
this pull request
Mar 31, 2025
…ultiple benchmarks (apache#14642) * Add support --mem-pool-type and --memory-limit options for all benchmarks * Add --sort-spill-reservation-bytes option
acking-you
pushed a commit
to acking-you/arrow-datafusion
that referenced
this pull request
Mar 31, 2025
…ultiple benchmarks (apache#14642) * Add support --mem-pool-type and --memory-limit options for all benchmarks * Add --sort-spill-reservation-bytes option
acking-you
pushed a commit
to acking-you/arrow-datafusion
that referenced
this pull request
Apr 7, 2025
…ultiple benchmarks (apache#14642) * Add support --mem-pool-type and --memory-limit options for all benchmarks * Add --sort-spill-reservation-bytes option
acking-you
pushed a commit
to acking-you/arrow-datafusion
that referenced
this pull request
Apr 7, 2025
…ultiple benchmarks (apache#14642) * Add support --mem-pool-type and --memory-limit options for all benchmarks * Add --sort-spill-reservation-bytes option
alamb
added a commit
that referenced
this pull request
Apr 8, 2025
* [draft] add shot circuit in BinaryExpr * refactor: add check_short_circuit function * refactor: change if condition to match * feat: Add support for --mem-pool-type and --memory-limit options to multiple benchmarks (#14642) * Add support --mem-pool-type and --memory-limit options for all benchmarks * Add --sort-spill-reservation-bytes option * Chore/Add additional FFI unit tests (#14802) * Add unit tests to FFI_ExecutionPlan * Add unit tests for FFI table source * Add round trip tests for volatility * Add unit tests for FFI insert op * Simplify string generation in unit test Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Fix drop of borrowed value --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Improve feature flag CI coverage `datafusion` and `datafusion-functions` (#15203) * add extend sql & docs * feat: Add support for --mem-pool-type and --memory-limit options to multiple benchmarks (#14642) * Add support --mem-pool-type and --memory-limit options for all benchmarks * Add --sort-spill-reservation-bytes option * Chore/Add additional FFI unit tests (#14802) * Add unit tests to FFI_ExecutionPlan * Add unit tests for FFI table source * Add round trip tests for volatility * Add unit tests for FFI insert op * Simplify string generation in unit test Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Fix drop of borrowed value --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Improve feature flag CI coverage `datafusion` and `datafusion-functions` (#15203) * fix: incorrect false judgment * add test * separate q6 to new PR * feat: Add support for --mem-pool-type and --memory-limit options to multiple benchmarks (#14642) * Add support --mem-pool-type and --memory-limit options for all benchmarks * Add --sort-spill-reservation-bytes option * Chore/Add additional FFI unit tests (#14802) * Add unit tests to FFI_ExecutionPlan * Add unit tests for FFI table source * Add round trip tests for volatility * Add unit tests for FFI insert op * Simplify string generation in unit test Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Fix drop of borrowed value --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Improve feature flag CI coverage `datafusion` and `datafusion-functions` (#15203) * feat: Add support for --mem-pool-type and --memory-limit options to multiple benchmarks (#14642) * Add support --mem-pool-type and --memory-limit options for all benchmarks * Add --sort-spill-reservation-bytes option * Chore/Add additional FFI unit tests (#14802) * Add unit tests to FFI_ExecutionPlan * Add unit tests for FFI table source * Add round trip tests for volatility * Add unit tests for FFI insert op * Simplify string generation in unit test Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Fix drop of borrowed value --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Improve feature flag CI coverage `datafusion` and `datafusion-functions` (#15203) * add benchmark for boolean_op * fix cargo doc * add binary_op bench * Better comments --------- Co-authored-by: Kristin Cowalcijk <bo@wherobots.com> Co-authored-by: Tim Saucer <timsaucer@gmail.com> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
nirnayroy
pushed a commit
to nirnayroy/datafusion
that referenced
this pull request
May 2, 2025
* [draft] add shot circuit in BinaryExpr * refactor: add check_short_circuit function * refactor: change if condition to match * feat: Add support for --mem-pool-type and --memory-limit options to multiple benchmarks (apache#14642) * Add support --mem-pool-type and --memory-limit options for all benchmarks * Add --sort-spill-reservation-bytes option * Chore/Add additional FFI unit tests (apache#14802) * Add unit tests to FFI_ExecutionPlan * Add unit tests for FFI table source * Add round trip tests for volatility * Add unit tests for FFI insert op * Simplify string generation in unit test Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Fix drop of borrowed value --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Improve feature flag CI coverage `datafusion` and `datafusion-functions` (apache#15203) * add extend sql & docs * feat: Add support for --mem-pool-type and --memory-limit options to multiple benchmarks (apache#14642) * Add support --mem-pool-type and --memory-limit options for all benchmarks * Add --sort-spill-reservation-bytes option * Chore/Add additional FFI unit tests (apache#14802) * Add unit tests to FFI_ExecutionPlan * Add unit tests for FFI table source * Add round trip tests for volatility * Add unit tests for FFI insert op * Simplify string generation in unit test Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Fix drop of borrowed value --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Improve feature flag CI coverage `datafusion` and `datafusion-functions` (apache#15203) * fix: incorrect false judgment * add test * separate q6 to new PR * feat: Add support for --mem-pool-type and --memory-limit options to multiple benchmarks (apache#14642) * Add support --mem-pool-type and --memory-limit options for all benchmarks * Add --sort-spill-reservation-bytes option * Chore/Add additional FFI unit tests (apache#14802) * Add unit tests to FFI_ExecutionPlan * Add unit tests for FFI table source * Add round trip tests for volatility * Add unit tests for FFI insert op * Simplify string generation in unit test Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Fix drop of borrowed value --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Improve feature flag CI coverage `datafusion` and `datafusion-functions` (apache#15203) * feat: Add support for --mem-pool-type and --memory-limit options to multiple benchmarks (apache#14642) * Add support --mem-pool-type and --memory-limit options for all benchmarks * Add --sort-spill-reservation-bytes option * Chore/Add additional FFI unit tests (apache#14802) * Add unit tests to FFI_ExecutionPlan * Add unit tests for FFI table source * Add round trip tests for volatility * Add unit tests for FFI insert op * Simplify string generation in unit test Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Fix drop of borrowed value --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Improve feature flag CI coverage `datafusion` and `datafusion-functions` (apache#15203) * add benchmark for boolean_op * fix cargo doc * add binary_op bench * Better comments --------- Co-authored-by: Kristin Cowalcijk <bo@wherobots.com> Co-authored-by: Tim Saucer <timsaucer@gmail.com> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
--memory-limitfor all benchmarking tools #14641.Rationale for this change
I had to run sort-tpch queries with memory limit when testing fixes for memory related issues, so I decide to add
--memory-limitoption for most of the benchmarking cli tools. I wish other developers could find it handy.What changes are included in this PR?
This PR adds 2 cli options
--memory-limit,--mem-pool-typeand--sort-spill-reservation-bytesto the following benchmarking tools:dfbenchsubcommands:sort,sort-tpch,clickbench,h2o,imdb,parquet-filtertpchimdbexternal_aggralready supports--memory-limit, it now accepts--mem-pool-type. The default value of--mem-pool-typeisfairso the behavior remains unchanged.Are these changes tested?
The changes were tested manually.
Are there any user-facing changes?
No. The benchmarking guide has not covered every option so hopefully the developers could find these options themselves using
--help.