Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

*: add explicit partition pruning to index joins (#32007) #32093

Merged
merged 16 commits into from
Mar 9, 2022

Conversation

mjonss
Copy link
Contributor

@mjonss mjonss commented Feb 2, 2022

What problem does this PR solve?

Issue Number: close #32007

Problem Summary:
Explicit partition selection SELECT ... FROM pt PARTITION(p) ... was not considered when buildTableReaderForIndexJoin().

What is changed and how it works?

Use the PartitionNames (if set) and do skip looking up contents in non-mentioned partitions.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Fixed explicit partition selection in subselects.

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Feb 2, 2022

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • djshow832
  • tiancaiamao

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added do-not-merge/needs-triage-completed release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. needs-cherry-pick-release-5.0 needs-cherry-pick-release-5.1 needs-cherry-pick-release-5.2 needs-cherry-pick-release-5.3 Type: Need cherry pick to release-5.3 needs-cherry-pick-release-5.4 Should cherry pick this PR to release-5.4 branch. and removed do-not-merge/needs-triage-completed labels Feb 2, 2022
@mjonss
Copy link
Contributor Author

mjonss commented Feb 2, 2022

/run-unit-test

2 similar comments
@hawkingrei
Copy link
Member

/run-unit-test

@hawkingrei
Copy link
Member

/run-unit-test

@sre-bot
Copy link
Contributor

sre-bot commented Feb 6, 2022

Copy link
Contributor

@djshow832 djshow832 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm curious: does only IndexJoin have this problem?

@mjonss
Copy link
Contributor Author

mjonss commented Feb 10, 2022

I'm curious: does only IndexJoin have this problem?

Great question! Not sure, but I just tried with Hash join and Merge join for the same query, and it does give the correct result.

select /*+ HASH_JOIN(t3) */ * from t3 where t3.a <> ALL (select t1.a from t1 partition (p0)) order by t3.a;
select /*+ MERGE_JOIN(t3) */ * from t3 where t3.a <> ALL (select t1.a from t1 partition (p0)) order by t3.a;

Notice that the EXPLAIN of the original query says it only uses partition:p0 which is wrong (i.e. the explain code uses the given partition (p0) but the index join does not implement it.

@djshow832 Which other types of joins needs to be tested?

@tiancaiamao tiancaiamao self-requested a review February 11, 2022 02:27
Comment on lines 3967 to 3969
if _, ok := explicitPartitionSelection[pid]; !ok {
continue
}
Copy link
Contributor

@djshow832 djshow832 Feb 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the partitions are already pruned by partitionProcessor.convertToIntSlice() indirectly through partitionPruning() in this code block.
Thus, I think the code branches where partitionPruning() is not called may have the problem.

@djshow832
Copy link
Contributor

I'm curious: does only IndexJoin have this problem?

Great question! Not sure, but I just tried with Hash join and Merge join for the same query, and it does give the correct result.

select /*+ HASH_JOIN(t3) */ * from t3 where t3.a <> ALL (select t1.a from t1 partition (p0)) order by t3.a;
select /*+ MERGE_JOIN(t3) */ * from t3 where t3.a <> ALL (select t1.a from t1 partition (p0)) order by t3.a;

Notice that the EXPLAIN of the original query says it only uses partition:p0 which is wrong (i.e. the explain code uses the given partition (p0) but the index join does not implement it.

@djshow832 Which other types of joins needs to be tested?

Not only joins but also other operators, which are hard to be tested fully. So I think the better way is to go through the code to check why buildTableReaderForIndexJoin missed it and how do other operators check it.

Copy link
Contributor

@tiancaiamao tiancaiamao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm afraid this is not the only place that missing the partition selection.
buildIndexReaderForIndexJoin / buildIndexLookupReaderForIndexJoin may have similiar issues...

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Feb 11, 2022
@tiancaiamao
Copy link
Contributor

I think the better way is to go through the code to check why buildTableReaderForIndexJoin missed it and how do other operators check it.

The code is too complex, that's the real root cause for this kind of bug.
As you can see, the code sketch is ..

if v.IsCommonHandle() {
    if len(lookUpContents) > 0 && keyColumnsIncludeAllPartitionColumns(lookUpContents[0].keyCols, pe) {
        // buggy branch
    } else {
        partitionPruning()
    }
} else {
   if len(lookUpContents) > 0 && keyColumnsIncludeAllPartitionColumns(lookUpContents[0].keyCols, pe) {
        // buggy branch
   } else {
        partitionPruning()
   }
}

@ti-chi-bot ti-chi-bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Feb 28, 2022
@mjonss mjonss marked this pull request as draft February 28, 2022 13:21
@ti-chi-bot ti-chi-bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 28, 2022
@ti-chi-bot ti-chi-bot removed the status/LGT1 Indicates that a PR has LGTM 1. label Mar 8, 2022
@djshow832
Copy link
Contributor

/merge

@ti-chi-bot ti-chi-bot added the status/LGT2 Indicates that a PR has LGTM 2. label Mar 8, 2022
@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: b7f8943

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Mar 8, 2022
@mjonss
Copy link
Contributor Author

mjonss commented Mar 8, 2022

/run-unit-test

@mjonss
Copy link
Contributor Author

mjonss commented Mar 8, 2022

/run-unit-test

@ti-chi-bot ti-chi-bot merged commit b9bd5a7 into pingcap:master Mar 9, 2022
ti-srebot pushed a commit to ti-srebot/tidb that referenced this pull request Mar 9, 2022
Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
@ti-srebot
Copy link
Contributor

cherry pick to release-5.0 in PR #32929

ti-srebot pushed a commit to ti-srebot/tidb that referenced this pull request Mar 9, 2022
Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
@ti-srebot
Copy link
Contributor

cherry pick to release-5.1 in PR #32930

ti-srebot pushed a commit to ti-srebot/tidb that referenced this pull request Mar 9, 2022
Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
@ti-srebot
Copy link
Contributor

cherry pick to release-5.2 in PR #32931

ti-srebot pushed a commit to ti-srebot/tidb that referenced this pull request Mar 9, 2022
Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
@ti-srebot
Copy link
Contributor

cherry pick to release-5.3 in PR #32932

ti-srebot pushed a commit to ti-srebot/tidb that referenced this pull request Mar 9, 2022
Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
@ti-srebot
Copy link
Contributor

cherry pick to release-5.4 in PR #32933

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs-cherry-pick-release-5.0 needs-cherry-pick-release-5.1 needs-cherry-pick-release-5.2 needs-cherry-pick-release-5.3 Type: Need cherry pick to release-5.3 needs-cherry-pick-release-5.4 Should cherry pick this PR to release-5.4 branch. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

partition as subquery with <> all got wrong result under dynamic pruning mode
7 participants