-
Notifications
You must be signed in to change notification settings - Fork 1.7k
Description
Is your feature request related to a problem or challenge?
While trying to test #7400 with datafusion-cli
I found I can't do it with datafuson-cli because datafusion-cli doesn't have a memory manager enabled.
Describe the solution you'd like
I would like to add two new new command line options to datafusion-cli
-m
/--mem-limit
that if set, would set the memory pool size limit. If unset no memory pool is used--mem-pool-type=<greedy|fair>
, defaults togreedy
that specifies the pool type: GreedyMemoryPool or FairSpillPool respectively
Examples of usage
# memory is not limited
datafusion-cli -c 'select 1, 2 from foo';
# run query with greedy memory pool set to use 10G
datafusion-cli --memory-limit 10G -c 'select 1, 2 from foo';
# run query with greedy memory pool set to use 10G
datafusion-cli -m 10G -c 'select 1, 2 from foo';
# run query with fair memory pool set to use 10G
datafusion-cli --pool-type=fair -m 10G -c 'select 1, 2 from foo';
See https://docs.rs/datafusion/latest/datafusion/execution/memory_pool/struct.FairSpillPool.html for more details
Describe alternatives you've considered
I also thought about setting the pools via SET
commands (like setting the target batch size). However, I don't think we should allow change memory limits via SQL because memory limits is likely not something a multi-tenant system would like to do . It should be setup before the session starts or by the runtime system, not the user in SQL
Additional context
Since this is well specified and is mostly an exercise in figuring out how datafusion-cli
works, I think this would make a good first project