remove hard coded partition count in ballista logicalplan deserialization

[xudong963]

Which issue does this PR close?

Closes ##972

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

[houqp]

cc @rdettai @yjshen

[houqp]

Looks like we only changed the function argument name, so it should be backwards compatible 👍

[rdettai]

Looks great! I still find the target_partitions a bit confusing. Could we maybe add more details to the comments in ExecutionConfig ? Thanks for your work!

[alamb]

I think this looks great -- thank you @xudong963 !

Adding some more doc strings would be useful as @rdettai but I think it can also be done as a follow on PR.

Thanks again for sticking with this

[xudong963]

Thanks for your help!
Looking forward to contributing more PRs.

[xudong963]

Could we maybe add more details to the comments in ExecutionConfig?

/// Number of partitions for query execution. Increasing partitions may increase concurrency.
/// The partition number can be much larger than the parallelism, then tasks for extra partitions will be queued.
/// So increasing partitions don't increase concurrency in the situation.

Please help me check if the comments are correct and detail @rdettai, I will add it in the next PR.

[rdettai]

I think that the current DataFusion default collection method does start all the tasks in parallel. I will try to go through all the places where the parameter is used to try to synthesize more precisely what it means.