doc: Add Join Physical Plan documentation, and configuration flag to benchmarks #18209
Conversation
|
@comphead This is for the documentation that was mentioned in the PiecewiseMergeJoin pull request |
2010YOUY01
left a comment
2010YOUY01
left a comment
There was a problem hiding this comment.
Nice doc, thank you.
I think it's better to move to https://datafusion.apache.org/user-guide/configs.html#tuning-guide, this should be a user guide instead of an internal doc.
| - When enabled, the physical planner may select PiecewiseMergeJoin if there is exactly one range | ||
| filter in the join condition. | ||
| - Piecewise Merge Join is faster than Nested Loop Join performance wise for single range filter | ||
| except for cases where it is joining two large tables (num_rows > 100,000) that are approximately |
There was a problem hiding this comment.
I think in theory PWMJ should still be faster, perhaps we have missed some detail in the implementation.
There was a problem hiding this comment.
A later todo may profile some classic cases for PWMJ to see the specific cost
I agree |
xudong963
left a comment
xudong963
left a comment
There was a problem hiding this comment.
I leave a ✅ here, and after moving the doc, it's good to go
github-actions
bot
added
documentation
Done! |
docs/source/user-guide/configs.md
Outdated
| Use the following command to update a configuration: | ||
|
|
||
| ``` | ||
| set datafusion.optimizer.<configuration_name> |
There was a problem hiding this comment.
would be nice to get a real example of setting the param?
There was a problem hiding this comment.
Oh I see confs are below, I think we can improve this nit?
There was a problem hiding this comment.
I provided an example right under
| - true: favors HashJoin for faster execution when sufficient memory is available. | ||
| - false: allows SortMergeJoin to be chosen when more memory-efficient execution is needed. | ||
|
|
||
| ### enable_piecewise_merge_join (bool, default = false) |
There was a problem hiding this comment.
btw you may also want include it in tpch cli utility, so people can test TPC queries with this kind of join
github-actions
bot
added
the
development-process
alamb
changed the title
alamb
changed the title
alamb
left a comment
alamb
left a comment
There was a problem hiding this comment.
Thank you @jonathanc-n @xudong963 and @comphead
NGA-TRAN
left a comment
NGA-TRAN
left a comment
There was a problem hiding this comment.
Thanks for the document, @jonathanc-n—great to know that Symmetric Hash Join and Piecewise Merge Join are available
…benchmarks (apache#18209) ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes apache#123` indicates that this PR will close issue apache#123. --> - Closes #. ## Rationale for this change Allow users to understand some decisions for when to change certain joins configurations. <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? Add readme to joins physical plan <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. -->
6 participants
Which issue does this PR close?
Rationale for this change
Allow users to understand some decisions for when to change certain joins configurations.
What changes are included in this PR?
Add readme to joins physical plan
Are these changes tested?
Are there any user-facing changes?