Interpreter: Serialize executors #4742

ywqzzy · 2022-04-24T05:26:53Z

What problem does this PR solve?

Issue Number: ref #4609

Problem Summary:

Currenly we don't have a serailizer of Executors that contains necessary and clean infomation.
The default serailizer of tipb::Executor contains too much info that may be hard to read.

What is changed and how it works?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

None

…alize_executor

ti-chi-bot · 2022-04-24T05:26:54Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

Lloyd-Pottiger
SeaRise

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

dbms/src/TestUtils/executorSerializer.h

dbms/src/TestUtils/executorSerializer.cpp

dbms/src/TestUtils/tests/gtest_mock_executors.cpp

dbms/src/TestUtils/executorSerializer.cpp

dbms/src/TestUtils/executorSerializer.h

dbms/src/TestUtils/tests/gtest_mock_executors.cpp

ywqzzy · 2022-04-24T10:46:40Z

/run-all-tests

dbms/src/TestUtils/executorSerializer.cpp

sre-bot · 2022-04-24T11:06:46Z

Coverage for changed files

Filename                                     Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Debug/astToExecutor.cpp                          503               210    58.25%          48                 9    81.25%        1327               571    56.97%         500               243    51.40%
TestUtils/InterpreterTestUtils.cpp                25                 6    76.00%           4                 2    50.00%          23                 9    60.87%           6                 3    50.00%
TestUtils/InterpreterTestUtils.h                   2                 0   100.00%           2                 0   100.00%           4                 0   100.00%           0                 0         -
TestUtils/executorSerializer.cpp                  96                34    64.58%          31                 1    96.77%         415                84    79.76%         108                36    66.67%
TestUtils/executorSerializer.h                     3                 0   100.00%           3                 0   100.00%           4                 0   100.00%           0                 0         -
TestUtils/mockExecutor.cpp                        57                10    82.46%          34                 7    79.41%         224                38    83.04%          28                 3    89.29%
TestUtils/mockExecutor.h                           5                 1    80.00%           5                 1    80.00%          14                 3    78.57%           0                 0         -
TestUtils/tests/gtest_mock_executors.cpp         234               171    26.92%          10                 0   100.00%         159                 0   100.00%          36                36     0.00%
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                            925               432    53.30%         137                20    85.40%        2170               705    67.51%         678               321    52.65%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18054      10148            43.79%    199976  100969       49.51%

full coverage report (for internal network access only)

dbms/src/TestUtils/tests/gtest_mock_executors.cpp

dbms/src/TestUtils/executorSerializer.cpp

ywqzzy · 2022-04-25T09:42:55Z

/run-all-tests

dbms/src/TestUtils/executorSerializer.cpp

sre-bot · 2022-04-25T10:22:42Z

Coverage for changed files

Filename                                     Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Debug/astToExecutor.cpp                          503               210    58.25%          48                 9    81.25%        1327               571    56.97%         500               243    51.40%
Flash/Coprocessor/DAGUtils.cpp                   351               219    37.61%          44                18    59.09%         628               388    38.22%         416               250    39.90%
Flash/Coprocessor/DAGUtils.h                       3                 0   100.00%           1                 0   100.00%          12                 0   100.00%           2                 0   100.00%
Flash/Statistics/ExchangeSenderImpl.cpp           17                17     0.00%           5                 5     0.00%          58                58     0.00%           8                 8     0.00%
TestUtils/InterpreterTestUtils.cpp                25                 6    76.00%           4                 2    50.00%          23                 9    60.87%           6                 3    50.00%
TestUtils/InterpreterTestUtils.h                   2                 0   100.00%           2                 0   100.00%           4                 0   100.00%           0                 0         -
TestUtils/executorSerializer.cpp                  66                13    80.30%          21                 1    95.24%         288                40    86.11%          52                11    78.85%
TestUtils/executorSerializer.h                     3                 0   100.00%           3                 0   100.00%           4                 0   100.00%           0                 0         -
TestUtils/mockExecutor.cpp                        57                10    82.46%          34                 7    79.41%         222                38    82.88%          28                 3    89.29%
TestUtils/mockExecutor.h                           5                 1    80.00%           5                 1    80.00%          14                 3    78.57%           0                 0         -
TestUtils/tests/gtest_mock_executors.cpp         234               171    26.92%          10                 0   100.00%         197                 0   100.00%          36                36     0.00%
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                           1266               647    48.89%         177                43    75.71%        2777              1107    60.14%        1048               554    47.14%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18055      10152            43.77%    200024  100952       49.53%

full coverage report (for internal network access only)

dbms/src/TestUtils/executorSerializer.h

…alize_executor

dbms/src/TestUtils/mockExecutor.cpp

dbms/src/TestUtils/executorSerializer.cpp

ywqzzy · 2022-04-26T05:11:25Z

/merge

ti-chi-bot · 2022-04-26T05:11:25Z

@ywqzzy: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2022-04-26T05:11:27Z

This pull request has been accepted and is ready to merge.

Commit hash: fbba86f

sre-bot · 2022-04-26T05:37:10Z

Coverage for changed files

Filename                                     Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Debug/astToExecutor.cpp                          503               210    58.25%          48                 9    81.25%        1327               571    56.97%         500               243    51.40%
Flash/Coprocessor/DAGUtils.cpp                   351               219    37.61%          44                18    59.09%         628               388    38.22%         416               250    39.90%
Flash/Coprocessor/DAGUtils.h                       3                 0   100.00%           1                 0   100.00%          12                 0   100.00%           2                 0   100.00%
Flash/Statistics/ExchangeSenderImpl.cpp           17                17     0.00%           5                 5     0.00%          58                58     0.00%           8                 8     0.00%
TestUtils/InterpreterTestUtils.cpp                25                 6    76.00%           4                 2    50.00%          21                 9    57.14%           6                 3    50.00%
TestUtils/InterpreterTestUtils.h                   2                 0   100.00%           2                 0   100.00%           4                 0   100.00%           0                 0         -
TestUtils/executorSerializer.cpp                  59                10    83.05%          23                 1    95.65%         282                35    87.59%          44                 9    79.55%
TestUtils/executorSerializer.h                     1                 0   100.00%           1                 0   100.00%           1                 0   100.00%           0                 0         -
TestUtils/mockExecutor.cpp                        57                 9    84.21%          34                 7    79.41%         222                38    82.88%          28                 2    92.86%
TestUtils/mockExecutor.h                           5                 1    80.00%           5                 1    80.00%          14                 3    78.57%           0                 0         -
TestUtils/tests/gtest_mock_executors.cpp         238               171    28.15%          10                 0   100.00%         205                 0   100.00%          36                36     0.00%
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                           1261               643    49.01%         177                43    75.71%        2774              1102    60.27%        1040               551    47.02%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18055      10152            43.77%    200021  100943       49.53%

full coverage report (for internal network access only)

ywqzzy added 4 commits April 18, 2022 20:57

update.

b287939

Merge branch 'master' of https://github.com/pingcap/tiflash into seri…

abc1dc7

…alize_executor

update.

4fb220d

update.

fc872ad

ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Apr 24, 2022

ywqzzy added 2 commits April 24, 2022 13:29

refine.

628f850

Merge branch 'master' into serialize_executor

823ac0c

ywqzzy requested review from SeaRise and Lloyd-Pottiger April 24, 2022 06:24

little refinement.

5fe993e

SeaRise reviewed Apr 24, 2022

View reviewed changes

SeaRise requested a review from JaySon-Huang April 24, 2022 06:53

SeaRise reviewed Apr 24, 2022

View reviewed changes

dbms/src/TestUtils/tests/gtest_mock_executors.cpp Outdated Show resolved Hide resolved

address comments.

bc2094c

SeaRise self-requested a review April 24, 2022 08:08

Lloyd-Pottiger reviewed Apr 24, 2022

View reviewed changes

dbms/src/TestUtils/executorSerializer.cpp Outdated Show resolved Hide resolved

dbms/src/TestUtils/executorSerializer.h Outdated Show resolved Hide resolved

ywqzzy added 2 commits April 24, 2022 16:33

update.

ad2c213

print sender schema.

f079911

SeaRise reviewed Apr 24, 2022

View reviewed changes

dbms/src/TestUtils/tests/gtest_mock_executors.cpp Outdated Show resolved Hide resolved

ywqzzy requested a review from Lloyd-Pottiger April 24, 2022 10:17

address comments.

1119e9c

ywqzzy requested a review from SeaRise April 24, 2022 10:26

SeaRise reviewed Apr 24, 2022

View reviewed changes

dbms/src/TestUtils/executorSerializer.cpp Outdated Show resolved Hide resolved

dbms/src/TestUtils/executorSerializer.cpp Outdated Show resolved Hide resolved

dbms/src/TestUtils/executorSerializer.cpp Outdated Show resolved Hide resolved

SeaRise self-requested a review April 24, 2022 11:32

Lloyd-Pottiger reviewed Apr 24, 2022

View reviewed changes

address comments.

84d58a3

Lloyd-Pottiger approved these changes Apr 25, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Apr 25, 2022

SeaRise reviewed Apr 25, 2022

View reviewed changes

SeaRise self-requested a review April 25, 2022 10:01

address comments. Refine format of columns.

d7f3881

ywqzzy force-pushed the serialize_executor branch from 6eb8918 to d7f3881 Compare April 25, 2022 10:34

SeaRise reviewed Apr 25, 2022

View reviewed changes

dbms/src/TestUtils/executorSerializer.h Outdated Show resolved Hide resolved

ywqzzy added 6 commits April 25, 2022 18:58

remove overdesgin SerializerContext.

9067dcf

Merge branch 'master' of https://github.com/pingcap/tiflash into seri…

7fb6353

…alize_executor

clean.

0fab9f6

clean.

8236f29

clean.

d86aaea

clean.

aa1ee61

ywqzzy requested a review from SeaRise April 25, 2022 12:00

SeaRise reviewed Apr 25, 2022

View reviewed changes

dbms/src/TestUtils/mockExecutor.cpp Show resolved Hide resolved

dbms/src/TestUtils/executorSerializer.cpp Outdated Show resolved Hide resolved

dbms/src/TestUtils/executorSerializer.cpp Outdated Show resolved Hide resolved

address comments.

a69100c

ywqzzy requested a review from SeaRise April 26, 2022 03:39

SeaRise approved these changes Apr 26, 2022

View reviewed changes

dbms/src/TestUtils/executorSerializer.cpp Outdated Show resolved Hide resolved

dbms/src/TestUtils/executorSerializer.cpp Outdated Show resolved Hide resolved

dbms/src/TestUtils/executorSerializer.cpp Outdated Show resolved Hide resolved

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Apr 26, 2022

address comments

fbba86f

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Apr 26, 2022

ti-chi-bot merged commit 5a0b5eb into pingcap:master Apr 26, 2022

ywqzzy mentioned this pull request Apr 26, 2022

Interpreter & Executor & MPP unit test framework #4609

Closed

54 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Interpreter: Serialize executors #4742

Interpreter: Serialize executors #4742

ywqzzy commented Apr 24, 2022 •

edited

Loading

ti-chi-bot commented Apr 24, 2022 •

edited

Loading

ywqzzy commented Apr 24, 2022

sre-bot commented Apr 24, 2022

ywqzzy commented Apr 25, 2022

sre-bot commented Apr 25, 2022

ywqzzy commented Apr 26, 2022

ti-chi-bot commented Apr 26, 2022

ti-chi-bot commented Apr 26, 2022

sre-bot commented Apr 26, 2022

Interpreter: Serialize executors #4742

Interpreter: Serialize executors #4742

Conversation

ywqzzy commented Apr 24, 2022 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Apr 24, 2022 • edited Loading

ywqzzy commented Apr 24, 2022

sre-bot commented Apr 24, 2022

Coverage for changed files

Coverage summary

ywqzzy commented Apr 25, 2022

sre-bot commented Apr 25, 2022

Coverage for changed files

Coverage summary

ywqzzy commented Apr 26, 2022

ti-chi-bot commented Apr 26, 2022

ti-chi-bot commented Apr 26, 2022

sre-bot commented Apr 26, 2022

Coverage for changed files

Coverage summary

ywqzzy commented Apr 24, 2022 •

edited

Loading

ti-chi-bot commented Apr 24, 2022 •

edited

Loading