executor: reuse chunk in hash join v2 during restoring #56936

xzhangxian1008 · 2024-10-29T05:12:34Z

What problem does this PR solve?

Issue Number: close #56828

Problem Summary:

What changed and how does it work?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

tiprow · 2024-10-29T05:12:54Z

Hi @xzhangxian1008. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

xzhangxian1008 · 2024-10-29T05:14:14Z

/cc @windtalker @XuHuaiyu

codecov · 2024-10-29T05:25:56Z

Codecov Report

Attention: Patch coverage is 83.09859% with 12 lines in your changes missing coverage. Please review.

Project coverage is 73.6575%. Comparing base (30069c5) to head (7ee28e8).
Report is 2 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #56936        +/-   ##
================================================
+ Coverage   73.1633%   73.6575%   +0.4941%     
================================================
  Files          1674       1674                
  Lines        461261     461669       +408     
================================================
+ Hits         337474     340054      +2580     
+ Misses       103041     100876      -2165     
+ Partials      20746      20739         -7

Flag	Coverage Δ
integration	`43.2952% <0.0000%> (?)`
unit	`72.3221% <83.0985%> (+0.0324%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.6910% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`45.9803% <ø> (-0.0271%)`	⬇️

XuHuaiyu · 2024-11-12T03:14:48Z

pkg/util/chunk/chunk_in_disk.go

 	return chk, nil
 }

+// FillChunk fills a Chunk from the DataInDiskByChunks by chkIdx.
+func (d *DataInDiskByChunks) FillChunk(chkIdx int, chk *Chunk) error {


Suggested change

func (d *DataInDiskByChunks) FillChunk(chkIdx int, chk *Chunk) error {

func (d *DataInDiskByChunks) FillChunk(srcChkIdx int, destChk *Chunk) error {

XuHuaiyu · 2024-11-12T03:16:41Z

pkg/executor/join/hash_join_spill_helper.go

@@ -81,6 +81,7 @@ type hashJoinSpillHelper struct {
 	spillTriggedInBuildingStageForTest           bool
 	spillTriggeredBeforeBuildingHashTableForTest bool
 	allPartitionsSpilledForTest                  bool
+	skipProbeInRestoreForTest                    bool


Why we need this?

Why we need this?

I find that one test case is not covered before and add it in this pr.

XuHuaiyu · 2024-11-12T03:21:58Z

pkg/executor/join/hash_join_v2.go

@@ -434,6 +436,7 @@ type BuildWorkerV2 struct {
 	HasNullableKey bool
 	WorkerID       uint
 	builder        *rowTableBuilder
+	restoredChk    *chunk.Chunk


restoredChkBuf ?

restoredChkBuf ?

done

XuHuaiyu · 2024-11-12T03:22:55Z

Can we get the heap profile to show the difference after this commit?

windtalker · 2024-11-12T06:08:26Z

pkg/executor/join/outer_join_spill_test.go

-		{true, leftKeys, rightKeys, leftTypes, rightTypes, []int{0, 1, 3, 4}, []int{0, 2, 3, 4}, otherCondition, []int{0}, []int{4}, []int64{5000000, 1700000, 6000000, 1500000, 10000}},
-		{false, leftKeys, rightKeys, leftTypes, rightTypes, []int{0, 1, 3, 4}, []int{0, 2, 3, 4}, otherCondition, []int{0}, []int{4}, []int64{5000000, 1700000, 6000000, 1500000, 10000}},
+		{true, leftKeys, rightKeys, leftTypes, rightTypes, []int{0, 1, 3, 4}, []int{0, 2, 3, 4}, otherCondition, []int{0}, []int{4}, []int64{5000000, 1700000, 6000000, 500000, 10000}},
+		{false, leftKeys, rightKeys, leftTypes, rightTypes, []int{0, 1, 3, 4}, []int{0, 2, 3, 4}, otherCondition, []int{0}, []int{4}, []int64{5000000, 1700000, 6000000, 500000, 10000}},


why change these?

why change these?

I find that one test case is not covered before and add it in this pr.

xzhangxian1008 · 2024-11-12T06:31:08Z

Can we get the heap profile to show the difference after this commit?

I will paste it in this pr.

windtalker

lgtm

xzhangxian1008 · 2024-11-14T06:28:50Z

/cc @XuHuaiyu

XuHuaiyu · 2024-11-15T02:51:37Z

pkg/executor/join/hash_join_v2.go

@@ -359,6 +360,8 @@ type ProbeWorkerV2 struct {
 	// We build individual joinProbe for each join worker when use chunk-based
 	// execution, to avoid the concurrency of joiner.chk and joiner.selected.
 	JoinProbe ProbeV2
+
+	restoredChk *chunk.Chunk


restoredChkBuf

ti-chi-bot · 2024-11-22T02:38:20Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: windtalker, XuHuaiyu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [XuHuaiyu,windtalker]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2024-11-22T02:38:23Z

[LGTM Timeline notifier]

Timeline:

2024-11-12 07:17:23.263774468 +0000 UTC m=+340605.454643465: ☑️ agreed by windtalker.
2024-11-22 02:38:22.450324405 +0000 UTC m=+172090.069978922: ☑️ agreed by XuHuaiyu.

xzhangxian1008 · 2024-11-22T02:38:55Z

/cherrypick release-8.5

ti-chi-bot · 2024-11-22T02:38:58Z

@xzhangxian1008: once the present PR merges, I will cherry-pick it on top of release-8.5 in the new PR and assign it to you.

In response to this:

/cherrypick release-8.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

xzhangxian1008 · 2024-11-22T02:39:17Z

/retest

tiprow · 2024-11-22T02:39:39Z

@xzhangxian1008: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tiprow · 2024-11-22T02:56:24Z

@xzhangxian1008: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
fast_test_tiprow	`851f998`	link	true	`/test fast_test_tiprow`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

ti-chi-bot · 2024-12-05T13:01:43Z

@xzhangxian1008: new pull request created to branch release-8.5: #58018.

In response to this:

/cherrypick release-8.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

xzhangxian1008 · 2024-12-05T13:05:25Z

/cherrypick release-8.5

ti-chi-bot · 2024-12-05T13:06:07Z

@xzhangxian1008: new pull request could not be created: failed to create pull request against pingcap/tidb#release-8.5 from head ti-chi-bot:cherry-pick-56936-to-release-8.5: status code 422 not one of [201], body: {"message":"Validation Failed","errors":[{"resource":"PullRequest","code":"custom","message":"A pull request already exists for ti-chi-bot:cherry-pick-56936-to-release-8.5."}],"documentation_url":"https://docs.github.com/rest/pulls/pulls#create-a-pull-request","status":"422"}

In response to this:

/cherrypick release-8.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

close #56828

xzhangxian1008 added 5 commits October 29, 2024 10:39

add reuse chk in chunkInDisk

01b8bb7

refine chunkInDisk

b703792

reuse build chunk

6ec9634

ok

1f44417

merge master

31e5c03

ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Oct 29, 2024

ti-chi-bot bot requested review from windtalker and XuHuaiyu October 29, 2024 05:14

xzhangxian1008 added 4 commits November 7, 2024 09:18

Merge branch 'master' into reuse-chk-hj-spill

db0d483

add test

b627e78

tweaking

6956b0b

tweaking

fec5f84

XuHuaiyu reviewed Nov 12, 2024

View reviewed changes

windtalker reviewed Nov 12, 2024

View reviewed changes

xzhangxian1008 requested review from XuHuaiyu and windtalker November 12, 2024 06:31

address comments

bba8434

windtalker approved these changes Nov 12, 2024

View reviewed changes

ti-chi-bot bot added approved needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Nov 12, 2024

XuHuaiyu reviewed Nov 15, 2024

View reviewed changes

xzhangxian1008 added 2 commits November 22, 2024 05:43

Merge branch 'master' into reuse-chk-hj-spill

697b8dc

rename

851f998

xzhangxian1008 requested a review from XuHuaiyu November 21, 2024 21:53

XuHuaiyu approved these changes Nov 22, 2024

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Nov 22, 2024

xzhangxian1008 added 2 commits December 5, 2024 17:09

Merge branch 'master' into reuse-chk-hj-spill

7c066a1

fix ci

7ee28e8

ti-chi-bot bot merged commit f0070f5 into pingcap:master Dec 5, 2024
22 checks passed

xzhangxian1008 deleted the reuse-chk-hj-spill branch December 5, 2024 13:01

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Dec 5, 2024

This is an automated cherry-pick of pingcap#56936

65f0008

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Dec 5, 2024

executor: reuse chunk in hash join v2 during restoring (#56936) #58018

Merged

13 tasks

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Dec 5, 2024

This is an automated cherry-pick of pingcap#56936

e277439

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot bot pushed a commit that referenced this pull request Dec 24, 2024

executor: reuse chunk in hash join v2 during restoring (#56936) (#58018)

ef0a693

close #56828

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

executor: reuse chunk in hash join v2 during restoring #56936

executor: reuse chunk in hash join v2 during restoring #56936

xzhangxian1008 commented Oct 29, 2024 •

edited

Loading

tiprow bot commented Oct 29, 2024

xzhangxian1008 commented Oct 29, 2024

codecov bot commented Oct 29, 2024 •

edited

Loading

XuHuaiyu Nov 12, 2024

xzhangxian1008 Nov 12, 2024

XuHuaiyu Nov 12, 2024

xzhangxian1008 Nov 12, 2024

XuHuaiyu Nov 12, 2024

xzhangxian1008 Nov 12, 2024

XuHuaiyu commented Nov 12, 2024

windtalker Nov 12, 2024

xzhangxian1008 Nov 12, 2024

xzhangxian1008 commented Nov 12, 2024

windtalker left a comment

xzhangxian1008 commented Nov 14, 2024

XuHuaiyu Nov 15, 2024

ti-chi-bot bot commented Nov 22, 2024

ti-chi-bot bot commented Nov 22, 2024

xzhangxian1008 commented Nov 22, 2024

ti-chi-bot commented Nov 22, 2024

xzhangxian1008 commented Nov 22, 2024

tiprow bot commented Nov 22, 2024

tiprow bot commented Nov 22, 2024

ti-chi-bot commented Dec 5, 2024

xzhangxian1008 commented Dec 5, 2024

ti-chi-bot commented Dec 5, 2024

	func (d DataInDiskByChunks) FillChunk(chkIdx int, chk Chunk) error {
	func (d DataInDiskByChunks) FillChunk(srcChkIdx int, destChk Chunk) error {

executor: reuse chunk in hash join v2 during restoring #56936

executor: reuse chunk in hash join v2 during restoring #56936

Conversation

xzhangxian1008 commented Oct 29, 2024 • edited Loading

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

tiprow bot commented Oct 29, 2024

xzhangxian1008 commented Oct 29, 2024

codecov bot commented Oct 29, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

XuHuaiyu commented Nov 12, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xzhangxian1008 commented Nov 12, 2024

windtalker left a comment

Choose a reason for hiding this comment

xzhangxian1008 commented Nov 14, 2024

Choose a reason for hiding this comment

ti-chi-bot bot commented Nov 22, 2024

ti-chi-bot bot commented Nov 22, 2024

[LGTM Timeline notifier]

xzhangxian1008 commented Nov 22, 2024

ti-chi-bot commented Nov 22, 2024

xzhangxian1008 commented Nov 22, 2024

tiprow bot commented Nov 22, 2024

tiprow bot commented Nov 22, 2024

ti-chi-bot commented Dec 5, 2024

xzhangxian1008 commented Dec 5, 2024

ti-chi-bot commented Dec 5, 2024

xzhangxian1008 commented Oct 29, 2024 •

edited

Loading

codecov bot commented Oct 29, 2024 •

edited

Loading