Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Planner: support window #5363

Merged
merged 15 commits into from
Jul 26, 2022

Conversation

SeaRise
Copy link
Contributor

@SeaRise SeaRise commented Jul 13, 2022

What problem does this PR solve?

Issue Number: ref #4739

Problem Summary:

What is changed and how it works?

  • move enableFineGrainedShuffleExtraInfo and enableFineGrainedShuffle from DAGContext to FineGrainedShuffle.h.
  • Let PhysicalExchangeSender and PhysicalExchangeReceiver support fine grained shuffle.
  • Add PhysicalWindow and PhysicalWindowSort.
  • Add PhysicalPlan.disableRestoreConcurrency to avoid restoreConcurrency after window_sort that disable FineGrainedShuffle.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

None

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Jul 13, 2022

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • guo-shaoge
  • mengxin9014

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jul 13, 2022
@SeaRise SeaRise mentioned this pull request Jul 13, 2022
13 tasks
@SeaRise
Copy link
Contributor Author

SeaRise commented Jul 13, 2022

/run-all-tests

@sre-bot
Copy link
Collaborator

sre-bot commented Jul 13, 2022

Coverage for changed files

Filename                                       Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Coprocessor/DAGContext.cpp                         116                79    31.90%          26                12    53.85%         182               113    37.91%          74                58    21.62%
Coprocessor/DAGContext.h                            40                14    65.00%          31                12    61.29%          88                34    61.36%          12                 6    50.00%
Coprocessor/DAGQueryBlockInterpreter.cpp           255                85    66.67%          33                 5    84.85%         599               139    76.79%         162                56    65.43%
Coprocessor/FineGrainedShuffle.h                     3                 0   100.00%           3                 0   100.00%           7                 0   100.00%           0                 0         -
Coprocessor/InterpreterUtils.cpp                    33                 3    90.91%           8                 0   100.00%         119                 6    94.96%          26                 5    80.77%
Mpp/ExchangeReceiver.cpp                           409               409     0.00%          35                35     0.00%         596               596     0.00%         240               240     0.00%
Planner/PhysicalPlan.cpp                            76                26    65.79%          11                 0   100.00%         114                13    88.60%          46                16    65.22%
Planner/PhysicalPlanNode.cpp                        12                 0   100.00%           8                 0   100.00%          46                 0   100.00%           4                 0   100.00%
Planner/PhysicalPlanNode.h                           7                 3    57.14%           7                 3    57.14%           7                 3    57.14%           0                 0         -
Planner/PlanType.cpp                                14                 3    78.57%           1                 0   100.00%          31                 6    80.65%          26                 3    88.46%
Planner/PlanType.h                                   3                 2    33.33%           3                 2    33.33%           8                 7    12.50%           0                 0         -
Planner/Planner.cpp                                 56                14    75.00%           7                 1    85.71%          65                12    81.54%          40                14    65.00%
Planner/plans/PhysicalExchangeReceiver.cpp          16                16     0.00%           5                 5     0.00%          45                45     0.00%           8                 8     0.00%
Planner/plans/PhysicalExchangeSender.cpp            33                33     0.00%           6                 6     0.00%          94                94     0.00%          14                14     0.00%
Planner/plans/PhysicalExchangeSender.h               1                 1     0.00%           1                 1     0.00%           1                 1     0.00%           0                 0         -
Planner/plans/PhysicalProjection.cpp                13                 0   100.00%           6                 0   100.00%          94                 6    93.62%           6                 0   100.00%
Planner/plans/PhysicalWindow.cpp                     8                 0   100.00%           5                 0   100.00%          45                 3    93.33%           2                 0   100.00%
Planner/plans/PhysicalWindow.h                       1                 0   100.00%           1                 0   100.00%           1                 0   100.00%           0                 0         -
Planner/plans/PhysicalWindowSort.cpp                16                 7    56.25%           4                 0   100.00%          30                 0   100.00%           6                 3    50.00%
Planner/plans/PhysicalWindowSort.h                   1                 0   100.00%           1                 0   100.00%           1                 0   100.00%           0                 0         -
Planner/tests/gtest_physical_plan.cpp              249               161    35.34%          11                 0   100.00%         217                 2    99.08%          52                41    21.15%
tests/gtest_planner_interpreter.cpp                258               133    48.45%           8                 0   100.00%         838                 0   100.00%          28                28     0.00%
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                             1620               989    38.95%         221                82    62.90%        3228              1080    66.54%         746               492    34.05%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18681      9500             49.15%    211275  95897        54.61%

full coverage report (for internal network access only)

@SeaRise SeaRise requested a review from guo-shaoge July 13, 2022 09:25
Co-authored-by: guo-shaoge <shaoge1994@163.com>
@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Jul 14, 2022
@SeaRise
Copy link
Contributor Author

SeaRise commented Jul 18, 2022

/run-all-tests

@SeaRise
Copy link
Contributor Author

SeaRise commented Jul 18, 2022

/run-unit-test

@sre-bot
Copy link
Collaborator

sre-bot commented Jul 18, 2022

Coverage for changed files

Filename                                       Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Coprocessor/DAGContext.cpp                         116                79    31.90%          26                12    53.85%         182               113    37.91%          74                58    21.62%
Coprocessor/DAGContext.h                            40                14    65.00%          31                12    61.29%          88                34    61.36%          12                 6    50.00%
Coprocessor/DAGQueryBlockInterpreter.cpp           255                85    66.67%          33                 5    84.85%         599               139    76.79%         162                56    65.43%
Coprocessor/FineGrainedShuffle.h                     3                 0   100.00%           3                 0   100.00%           7                 0   100.00%           0                 0         -
Mpp/ExchangeReceiver.cpp                           409               409     0.00%          35                35     0.00%         596               596     0.00%         240               240     0.00%
Planner/PhysicalPlan.cpp                            76                26    65.79%          11                 0   100.00%         114                13    88.60%          46                16    65.22%
Planner/PhysicalPlanNode.cpp                        12                 0   100.00%           8                 0   100.00%          46                 0   100.00%           4                 0   100.00%
Planner/PhysicalPlanNode.h                           7                 3    57.14%           7                 3    57.14%           7                 3    57.14%           0                 0         -
Planner/PlanType.cpp                                14                 3    78.57%           1                 0   100.00%          31                 6    80.65%          26                 3    88.46%
Planner/PlanType.h                                   3                 2    33.33%           3                 2    33.33%           8                 7    12.50%           0                 0         -
Planner/Planner.cpp                                 64                14    78.12%           9                 1    88.89%          83                12    85.54%          50                15    70.00%
Planner/plans/PhysicalExchangeReceiver.cpp          16                16     0.00%           5                 5     0.00%          46                46     0.00%           8                 8     0.00%
Planner/plans/PhysicalExchangeSender.cpp            33                33     0.00%           6                 6     0.00%          94                94     0.00%          14                14     0.00%
Planner/plans/PhysicalExchangeSender.h               1                 1     0.00%           1                 1     0.00%           1                 1     0.00%           0                 0         -
Planner/plans/PhysicalProjection.cpp                13                 0   100.00%           6                 0   100.00%          94                 6    93.62%           6                 0   100.00%
Planner/plans/PhysicalWindow.cpp                     8                 0   100.00%           5                 0   100.00%          45                 3    93.33%           2                 0   100.00%
Planner/plans/PhysicalWindow.h                       1                 0   100.00%           1                 0   100.00%           1                 0   100.00%           0                 0         -
Planner/plans/PhysicalWindowSort.cpp                16                 7    56.25%           4                 0   100.00%          30                 0   100.00%           6                 3    50.00%
Planner/plans/PhysicalWindowSort.h                   1                 0   100.00%           1                 0   100.00%           1                 0   100.00%           0                 0         -
Planner/tests/gtest_physical_plan.cpp              249               161    35.34%          11                 0   100.00%         217                 2    99.08%          52                41    21.15%
tests/gtest_planner_interpreter.cpp                258               133    48.45%           8                 0   100.00%         838                 0   100.00%          28                28     0.00%
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                             1595               986    38.18%         215                82    61.86%        3128              1075    65.63%         730               488    33.15%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18770      9509             49.34%    214399  95992        55.23%

full coverage report (for internal network access only)

@mengxin9014
Copy link
Contributor

I suggest adding a few unit tests with multi-stream.

@mengxin9014
Copy link
Contributor

others LGTM.

@SeaRise
Copy link
Contributor Author

SeaRise commented Jul 25, 2022

I suggest adding a few unit tests with multi-stream.

wait for #5464 merged and merge to planner_refactory,

@SeaRise
Copy link
Contributor Author

SeaRise commented Jul 25, 2022

I suggest adding a few unit tests with multi-stream.

wait for #5464 merged and merge to planner_refactory,

I has port the test cases from #5464.

@SeaRise
Copy link
Contributor Author

SeaRise commented Jul 25, 2022

/run-all-tests

@sre-bot
Copy link
Collaborator

sre-bot commented Jul 25, 2022

Coverage for changed files

Filename                                       Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Coprocessor/DAGContext.cpp                         116                79    31.90%          26                12    53.85%         182               113    37.91%          74                58    21.62%
Coprocessor/DAGContext.h                            40                14    65.00%          31                12    61.29%          88                34    61.36%          12                 6    50.00%
Coprocessor/DAGQueryBlockInterpreter.cpp           255                83    67.45%          33                 5    84.85%         601               134    77.70%         162                54    66.67%
Coprocessor/FineGrainedShuffle.h                     3                 0   100.00%           3                 0   100.00%           7                 0   100.00%           0                 0         -
Mpp/ExchangeReceiver.cpp                           409               409     0.00%          35                35     0.00%         596               596     0.00%         240               240     0.00%
Planner/PhysicalPlan.cpp                           108                32    70.37%          12                 0   100.00%         166                25    84.94%          72                26    63.89%
Planner/PhysicalPlanNode.cpp                        12                 0   100.00%           8                 0   100.00%          46                 0   100.00%           4                 0   100.00%
Planner/PhysicalPlanNode.h                           7                 1    85.71%           7                 1    85.71%           7                 1    85.71%           0                 0         -
Planner/PlanType.cpp                                17                 4    76.47%           1                 0   100.00%          37                 8    78.38%          32                 4    87.50%
Planner/PlanType.h                                   3                 1    66.67%           3                 1    66.67%           8                 4    50.00%           0                 0         -
Planner/Planner.cpp                                 52                 7    86.54%           8                 0   100.00%          79                 2    97.47%          38                 7    81.58%
Planner/plans/PhysicalExchangeReceiver.cpp          16                16     0.00%           5                 5     0.00%          45                45     0.00%           8                 8     0.00%
Planner/plans/PhysicalExchangeSender.cpp            33                33     0.00%           6                 6     0.00%          94                94     0.00%          14                14     0.00%
Planner/plans/PhysicalExchangeSender.h               1                 1     0.00%           1                 1     0.00%           1                 1     0.00%           0                 0         -
Planner/plans/PhysicalProjection.cpp                13                 0   100.00%           6                 0   100.00%          94                 6    93.62%           6                 0   100.00%
Planner/plans/PhysicalWindow.cpp                     8                 0   100.00%           5                 0   100.00%          46                 3    93.48%           2                 0   100.00%
Planner/plans/PhysicalWindow.h                       1                 0   100.00%           1                 0   100.00%           1                 0   100.00%           0                 0         -
Planner/plans/PhysicalWindowSort.cpp                16                 7    56.25%           4                 0   100.00%          30                 0   100.00%           6                 3    50.00%
Planner/plans/PhysicalWindowSort.h                   1                 0   100.00%           1                 0   100.00%           1                 0   100.00%           0                 0         -
Planner/tests/gtest_physical_plan.cpp              303               199    34.32%          14                 0   100.00%         348                 3    99.14%          66                50    24.24%
tests/gtest_planner_interpreter.cpp                258               133    48.45%           8                 0   100.00%         850                 0   100.00%          28                28     0.00%
tests/gtest_window_executor.cpp                    112                31    72.32%           4                 0   100.00%         152                 0   100.00%          34                16    52.94%
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                             1784              1050    41.14%         222                78    64.86%        3479              1069    69.27%         798               514    35.59%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18985      9481             50.06%    216789  95832        55.79%

full coverage report (for internal network access only)

@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jul 26, 2022
@SeaRise
Copy link
Contributor Author

SeaRise commented Jul 26, 2022

/merge

@ti-chi-bot
Copy link
Member

@SeaRise: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 68eccd7

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jul 26, 2022
@sre-bot
Copy link
Collaborator

sre-bot commented Jul 26, 2022

Coverage for changed files

Filename                                       Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Coprocessor/DAGContext.cpp                         116                79    31.90%          26                12    53.85%         182               113    37.91%          74                58    21.62%
Coprocessor/DAGContext.h                            40                14    65.00%          31                12    61.29%          88                34    61.36%          12                 6    50.00%
Coprocessor/DAGQueryBlockInterpreter.cpp           255                83    67.45%          33                 5    84.85%         601               134    77.70%         162                54    66.67%
Coprocessor/FineGrainedShuffle.h                     3                 0   100.00%           3                 0   100.00%           7                 0   100.00%           0                 0         -
Mpp/ExchangeReceiver.cpp                           409               409     0.00%          35                35     0.00%         596               596     0.00%         240               240     0.00%
Planner/PhysicalPlan.cpp                           108                32    70.37%          12                 0   100.00%         166                25    84.94%          72                26    63.89%
Planner/PhysicalPlanNode.cpp                        12                 0   100.00%           8                 0   100.00%          46                 0   100.00%           4                 0   100.00%
Planner/PhysicalPlanNode.h                           7                 1    85.71%           7                 1    85.71%           7                 1    85.71%           0                 0         -
Planner/PlanType.cpp                                17                 4    76.47%           1                 0   100.00%          37                 8    78.38%          32                 4    87.50%
Planner/PlanType.h                                   3                 1    66.67%           3                 1    66.67%           8                 4    50.00%           0                 0         -
Planner/Planner.cpp                                 52                 7    86.54%           8                 0   100.00%          79                 2    97.47%          38                 7    81.58%
Planner/plans/PhysicalExchangeReceiver.cpp          16                16     0.00%           5                 5     0.00%          45                45     0.00%           8                 8     0.00%
Planner/plans/PhysicalExchangeSender.cpp            33                33     0.00%           6                 6     0.00%          94                94     0.00%          14                14     0.00%
Planner/plans/PhysicalExchangeSender.h               1                 1     0.00%           1                 1     0.00%           1                 1     0.00%           0                 0         -
Planner/plans/PhysicalProjection.cpp                13                 0   100.00%           6                 0   100.00%          94                 6    93.62%           6                 0   100.00%
Planner/plans/PhysicalWindow.cpp                     8                 0   100.00%           5                 0   100.00%          46                 3    93.48%           2                 0   100.00%
Planner/plans/PhysicalWindow.h                       1                 0   100.00%           1                 0   100.00%           1                 0   100.00%           0                 0         -
Planner/plans/PhysicalWindowSort.cpp                16                 7    56.25%           4                 0   100.00%          30                 0   100.00%           6                 3    50.00%
Planner/plans/PhysicalWindowSort.h                   1                 0   100.00%           1                 0   100.00%           1                 0   100.00%           0                 0         -
Planner/tests/gtest_physical_plan.cpp              303               199    34.32%          14                 0   100.00%         348                 3    99.14%          66                50    24.24%
tests/gtest_planner_interpreter.cpp                258               133    48.45%           8                 0   100.00%         850                 0   100.00%          28                28     0.00%
tests/gtest_window_executor.cpp                    112                31    72.32%           4                 0   100.00%         152                 0   100.00%          34                16    52.94%
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                             1784              1050    41.14%         222                78    64.86%        3479              1069    69.27%         798               514    35.59%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18985      9479             50.07%    216789  95768        55.82%

full coverage report (for internal network access only)

@ti-chi-bot ti-chi-bot merged commit d9bcbde into pingcap:planner_refactory Jul 26, 2022
@SeaRise SeaRise deleted the new_support_window branch January 11, 2023 09:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants