executor: support left outer semi join for hash join v2 #57053

wshwsh12 · 2024-11-01T03:29:25Z

What problem does this PR solve?

Issue Number: ref #53127

Problem Summary:

What changed and how does it work?

Support left outer semi join for hash join v2.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

tiprow · 2024-11-01T03:29:45Z

Hi @wshwsh12. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

codecov · 2024-11-01T06:29:16Z

Codecov Report

Attention: Patch coverage is 82.53275% with 40 lines in your changes missing coverage. Please review.

Project coverage is 58.1168%. Comparing base (c39f46f) to head (71d464d).
Report is 56 commits behind head on master.

Additional details and impacted files

@@                Coverage Diff                @@
##             master     #57053         +/-   ##
=================================================
- Coverage   72.9983%   58.1168%   -14.8815%     
=================================================
  Files          1657       1824        +167     
  Lines        457724     659502     +201778     
=================================================
+ Hits         334131     383282      +49151     
- Misses       103066     251034     +147968     
- Partials      20527      25186       +4659

Flag	Coverage Δ
integration	`39.6730% <60.2620%> (?)`
unit	`73.5084% <82.8193%> (+1.2285%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`55.2602% <ø> (+2.3124%)`	⬆️
parser	`∅ <ø> (∅)`
br	`63.1758% <ø> (+17.4965%)`	⬆️

wshwsh12 · 2024-11-04T08:05:50Z

/retest

tiprow · 2024-11-04T08:06:12Z

@wshwsh12: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

wshwsh12 · 2024-11-04T08:42:11Z

/retest

tiprow · 2024-11-04T08:42:31Z

@wshwsh12: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

wshwsh12 · 2024-11-04T08:48:34Z

/check_dev_2

wshwsh12 · 2024-11-04T08:58:25Z

/retest

tiprow · 2024-11-04T08:58:47Z

@wshwsh12: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

windtalker · 2024-11-05T05:35:54Z

pkg/executor/join/left_outer_semi_join_probe.go

+
+type leftOuterSemiJoinProbe struct {
+	baseJoinProbe
+	// build/probe side used columns and offset in result chunk


Looks like the comments can not explain the code clearly?

windtalker · 2024-11-05T05:53:37Z

pkg/executor/join/base_join_probe.go

@@ -747,6 +747,8 @@ func NewJoinProbe(ctx *HashJoinCtxV2, workID uint, joinType logicalop.JoinType,
 		return newOuterJoinProbe(base, !rightAsBuildSide, rightAsBuildSide)
 	case logicalop.RightOuterJoin:
 		return newOuterJoinProbe(base, rightAsBuildSide, rightAsBuildSide)
+	case logicalop.LeftOuterSemiJoin:


check rightAsBuildSide is always true here?

windtalker · 2024-11-05T06:03:42Z

pkg/executor/join/left_outer_semi_join_probe.go

+
+func (j *leftOuterSemiJoinProbe) buildResult(chk *chunk.Chunk, startProbeRow int) {
+	selected := make([]bool, j.chunkRows)
+	for i := startProbeRow; i < j.currentProbeRow; i++ {


if chk is empty, and startProbeRow == 0 && j.currentProbeRow == j.chunkRows && j.currentChunk.sel() == nil is true, can we use shallow copy instead of deep copy?

windtalker · 2024-11-05T06:07:29Z

pkg/executor/join/left_outer_semi_join_probe.go

+	}
+
+	if j.ctx.hasOtherCondition() {
+		err = j.probeForInnerSideBuildWithOtherCondition(joinResult.chk, joinedChk, sqlKiller)


seems no need to add ForInnerSideBuild since it always use inner side build

windtalker · 2024-11-05T06:07:39Z

pkg/executor/join/left_outer_semi_join_probe.go

+	if j.ctx.hasOtherCondition() {
+		err = j.probeForInnerSideBuildWithOtherCondition(joinResult.chk, joinedChk, sqlKiller)
+	} else {
+		err = j.probeForInnerSideBuildWithoutOtherCondition(joinResult.chk, joinedChk, remainCap, sqlKiller)


windtalker · 2024-11-05T07:08:49Z

pkg/executor/join/left_outer_semi_join_probe.go

+
+	// isMatchedRows marks whether the left side row is matched
+	isMatchedRows []bool
+	// isNullRows marks whether the left side row is null


Suggested change

// isNullRows marks whether the left side row is null

// isNullRows marks whether the left side row match result is null

windtalker · 2024-11-05T07:09:45Z

pkg/executor/join/left_outer_semi_join_probe.go

+		baseJoinProbe:           base,
+		processedProbeRowIdxSet: make(map[int]struct{}),
+	}
+	probe.leftColUsed = base.lUsed


why copy this field, you can use base.lUsed directly?

windtalker · 2024-11-05T07:45:21Z

pkg/executor/join/left_outer_semi_join_probe.go

+	}
+
+	if j.currentProbeRow == j.chunkRows && len(j.processedProbeRowIdxSet) == 0 {
+		j.buildResult(chk, 0)


Looks like it would exceed chk's capacity if the chk is not empty?

windtalker · 2024-11-06T09:42:48Z

pkg/executor/join/left_outer_semi_join_probe.go

+			j.processedProbeRowIdxQueue.Push(i)
+		}
+	}
+	return nil


looks like L56-L69 is exactly the same as L78-91, I think we can reuse the code

windtalker · 2024-11-07T00:32:32Z

pkg/executor/join/left_outer_semi_join_probe.go

+	}
+	if j.ctx.hasOtherCondition() {
+		j.processedProbeRowIdxQueue.Clear()
+		for i := 0; i < j.chunkRows; i++ {


Since matchedRowsHeaders is already set in base.SetChunkForProbe, I think we can only add the rows that matchedRowsHeaders[rows] != 0 to processedProbeIdxQueue.

windtalker · 2024-11-07T00:33:58Z

pkg/executor/join/left_outer_semi_join_probe.go

+	isNulls []bool
+
+	// used in other condition to record which rows need to be processed
+	processedProbeRowIdxQueue *queue.Queue[int]


maybe unFinishedProbeRowIdxQueue is a more self-explain name?

windtalker · 2024-11-07T00:34:33Z

pkg/executor/join/left_outer_semi_join_probe.go

+	}
+	if j.ctx.hasOtherCondition() {
+		j.processedProbeRowIdxQueue.Clear()
+		for i := 0; i < j.chunkRows; i++ {


windtalker · 2024-11-07T00:44:27Z

pkg/executor/join/left_outer_semi_join_probe.go

+}
+
+func (j *leftOuterSemiJoinProbe) concatenateProbeAndBuildRows(joinedChk *chunk.Chunk, sqlKiller *sqlkiller.SQLKiller) error {
+	joinedChkRemainCap := joinedChk.Capacity()


Suggested change

joinedChkRemainCap := joinedChk.Capacity()

joinedChkRemainCap := joinedChk.Capacity() - joinedChk.NumRows()

windtalker · 2024-11-07T00:47:11Z

pkg/executor/join/left_outer_semi_join_probe_test.go

+	}
+}
+
+func TestLeftOuterSemiJoinSpillBasic(t *testing.T) {


I think we need add 3 tests for spill

basic tests

with other condition

with other condition and sel array.

windtalker · 2024-11-07T00:47:43Z

pkg/executor/join/left_outer_semi_join_probe_test.go

+
+	err := failpoint.Enable("github.com/pingcap/tidb/pkg/executor/join/slowWorkers", `return(true)`)
+	require.NoError(t, err)
+	defer failpoint.Disable("github.com/pingcap/tidb/pkg/executor/join/slowWorkers")


I think basic test don't need to enable slow workers?

windtalker · 2024-11-07T00:53:10Z

pkg/executor/join/left_outer_semi_join_probe.go

+	if startProbeRow == 0 && j.currentProbeRow == j.chunkRows && j.currentChunk.Sel() == nil && chk.NumRows() == 0 && len(j.spilledIdx) == 0 {
+		// TODO: Can do a shallow copy by directly copying the Column pointers
+		for index, colIndex := range j.lUsed {
+			chk.SetCol(index, j.currentChunk.Column(colIndex).CopyConstruct(nil))


why not pass chk.Column(index) to CopyConstruct?

wshwsh12 · 2024-11-07T05:49:49Z

/retest

tiprow · 2024-11-07T05:50:10Z

@wshwsh12: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

windtalker

lgtm

xzhangxian1008

LGTM

ti-chi-bot · 2024-11-08T08:13:37Z

[LGTM Timeline notifier]

Timeline:

2024-11-07 06:26:57.09115944 +0000 UTC m=+1108729.930314989: ☑️ agreed by windtalker.
2024-11-08 08:13:36.449571081 +0000 UTC m=+1201529.288726628: ☑️ agreed by xzhangxian1008.

ti-chi-bot · 2024-11-11T06:27:13Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: AilinKid, windtalker, xzhangxian1008

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [AilinKid,windtalker]
~~pkg/expression/OWNERS~~ [windtalker]
~~pkg/planner/OWNERS~~ [AilinKid]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

wshwsh12 · 2024-11-11T07:31:29Z

/retest

tiprow · 2024-11-11T07:31:52Z

@wshwsh12: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

windtalker · 2024-11-13T01:41:24Z

/test unit-test

tiprow · 2024-11-13T01:41:45Z

@windtalker: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/test unit-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

windtalker · 2024-11-13T04:13:36Z

/test unit-test

tiprow · 2024-11-13T04:14:03Z

@windtalker: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/test unit-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

ref pingcap#53127 handle key-too-large error from MemBuffer Signed-off-by: you06 <you1474600@gmail.com> test MemBuffer's oversize error to tidb error Signed-off-by: you06 <you1474600@gmail.com> update errdoc Signed-off-by: you06 <you1474600@gmail.com>

wshwsh12 added 8 commits October 16, 2024 17:05

temp

e60b755

temp

9354e7c

Merge remote-tracking branch 'upstream/master' into left-outer-semi-join

e920c41

fix

58f5bd5

fix without cond

5cf401a

fix all

3f24021

fix null

83407ae

fix null

07a01d3

ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. sig/planner SIG: Planner size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Nov 1, 2024

wshwsh12 added 2 commits November 1, 2024 11:37

bazel

5427626

fix lint

949972a

Merge remote-tracking branch 'upstream/master' into left-outer-semi-join

2233dbf

wshwsh12 force-pushed the left-outer-semi-join branch from e340196 to 11e7453 Compare November 1, 2024 08:53

optimize

ec6ff80

wshwsh12 force-pushed the left-outer-semi-join branch from 11e7453 to ec6ff80 Compare November 1, 2024 09:15

fix

2134dd0

wshwsh12 added 2 commits November 4, 2024 17:15

Merge remote-tracking branch 'upstream/master' into left-outer-semi-join

e3c7145

fix

4155bb7

windtalker reviewed Nov 5, 2024

View reviewed changes

wshwsh12 added 5 commits November 6, 2024 15:45

address comments

3dde1a4

Merge remote-tracking branch 'upstream/master' into left-outer-semi-join

cc826fa

add SetRestoredChunkForProbe

d7582bf

spill

0d9c53d

add spill ut

f554c4d

wshwsh12 requested a review from windtalker November 6, 2024 16:41

windtalker reviewed Nov 7, 2024

View reviewed changes

add ut

71d464d

wshwsh12 requested a review from windtalker November 7, 2024 06:01

windtalker approved these changes Nov 7, 2024

View reviewed changes

ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Nov 7, 2024

xzhangxian1008 approved these changes Nov 8, 2024

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Nov 8, 2024

AilinKid approved these changes Nov 11, 2024

View reviewed changes

ti-chi-bot bot added the approved label Nov 11, 2024

ti-chi-bot bot merged commit 8832684 into pingcap:master Nov 13, 2024
23 checks passed

	// isNullRows marks whether the left side row is null
	// isNullRows marks whether the left side row match result is null

	joinedChkRemainCap := joinedChk.Capacity()
	joinedChkRemainCap := joinedChk.Capacity() - joinedChk.NumRows()

executor: support left outer semi join for hash join v2 #57053

executor: support left outer semi join for hash join v2 #57053

Conversation

wshwsh12 commented Nov 1, 2024

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

tiprow bot commented Nov 1, 2024

codecov bot commented Nov 1, 2024 • edited Loading

Codecov Report

wshwsh12 commented Nov 4, 2024

tiprow bot commented Nov 4, 2024

wshwsh12 commented Nov 4, 2024

tiprow bot commented Nov 4, 2024

wshwsh12 commented Nov 4, 2024

wshwsh12 commented Nov 4, 2024

tiprow bot commented Nov 4, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wshwsh12 commented Nov 7, 2024

tiprow bot commented Nov 7, 2024

windtalker left a comment

Choose a reason for hiding this comment

xzhangxian1008 left a comment

Choose a reason for hiding this comment

ti-chi-bot bot commented Nov 8, 2024

[LGTM Timeline notifier]

ti-chi-bot bot commented Nov 11, 2024

wshwsh12 commented Nov 11, 2024

tiprow bot commented Nov 11, 2024

windtalker commented Nov 13, 2024

tiprow bot commented Nov 13, 2024

windtalker commented Nov 13, 2024

tiprow bot commented Nov 13, 2024

codecov bot commented Nov 1, 2024 •

edited

Loading