plan: support `?` in Order By / Group By / Limit Offset clauses #8206

dbjoa · 2018-11-06T09:09:36Z

What problem does this PR solve?

Fix #8153

What is changed and how it works?

Modify parser.ByItem to support ast.ParamMarkerExpr
The prepared statements having parameters in OrderByClause or GroupByClause aren't cached

Check List

Tests

Unit test

Code changes

Has exported function/method change
Has persistent data change (i.e., parser)

Side effects

No

Related changes

Need to update the parser repository (i.e., the parser PR #27)

This change is

sre-bot · 2018-11-06T09:09:40Z

Hi contributor, thanks for your PR.

This patch needs to be approved by someone of admins. They should reply with "/ok-to-test" to accept this PR for running test automatically.

planner/core/expression_rewriter.go

zz-jason · 2018-11-06T09:26:39Z

@winoros @eurekaka PTAL

zz-jason · 2018-11-06T09:31:34Z

@dbjoa Would you file another PR to make this order by statement cacheable?

dbjoa · 2018-11-06T10:03:23Z

@zz-jason Yes, I would.

dbjoa · 2018-11-06T10:14:11Z

@zz-jason How can we run the CI test (i.e., run-all-tests) based on the parser PR #20 ?

zz-jason · 2018-11-06T10:27:56Z

@iamxy Could you help us with this?

How can we run the CI test (i.e., run-all-tests) based on the parser PR pingcap/parser#20 ?

planner/core/logical_plan_builder.go

planner/core/expression_rewriter.go

tiancaiamao · 2018-11-07T06:40:19Z

Hi, @dbjoa
pingcap/parser#20 broke the CI
The root problem is that, update TiDB and parser should be together, we didn't do it well during the migration to Go module.

Here I add some instructions on how to update parser for TiDB:
pingcap/parser#23

You can follow that.
Any suggestions are welcome, thanks.

dbjoa · 2018-11-07T10:24:13Z

/run-all-tests

dbjoa · 2018-11-07T11:00:17Z

@eurekaka
Would you let me know the schema and queries of the failed CI tests?

eurekaka · 2018-11-08T02:17:07Z

@dbjoa the log indicates it is the newly added case that fails:

FAIL: prepared_test.go:571: testSuite.TestPreparedIssue8153

prepared_test.go:602:
    c.Assert(err.Error(), Equals, "[planner:1054]Unknown column '?' in 'order clause'")
... obtained string = "[planner:1054]Unknown column '3' in 'order clause'"
... expected string = "[planner:1054]Unknown column '?' in 'order clause'"

dbjoa · 2018-11-08T06:40:20Z

/run-all-tests

dbjoa · 2018-11-08T07:18:52Z

/run-all-tests

shenli · 2018-11-08T07:19:17Z

/run-all-tests

zz-jason · 2018-11-15T02:53:32Z

@dbjoa Could you update go.mod to use the latest tidb parser?

dbjoa · 2018-11-15T03:24:42Z

@zz-jason I've updated the PR to use the latest parser.

dbjoa · 2018-11-15T05:02:10Z

/run-all-tests

dbjoa · 2018-11-15T05:08:30Z

/run-all-tests

planner/core/logical_plan_builder.go

dbjoa · 2018-11-15T06:48:44Z

/run-all-tests

dbjoa · 2018-11-30T23:22:39Z

I've updated PR to remove InPrepare flag by setting the default parameter values as NULL.
@eurekaka @zz-jason PTAL

eurekaka

LGTM

expression/util.go

winoros · 2018-12-03T05:37:25Z

expression/util.go

+	}
+	value, err := GetParamExpression(ctx, v.P.(*driver.ParamMarkerExpr))
+	if err != nil {
+		return 0, true, errors.Trace(err)


Once it's traced, we don't need to trace it anymore.
So here we can just return err.

Thank you for your detailed comments. The updated PR will address the issue.

crazycs520 · 2018-12-03T07:00:27Z

/run-all-tests

dbjoa · 2018-12-03T07:45:55Z

/run-integration-ddl-test

zz-jason · 2018-12-03T09:43:43Z

expression/util.go

@@ -526,3 +527,55 @@ func GetParamExpression(ctx sessionctx.Context, v *driver.ParamMarkerExpr, useCa
 	}
 	return value, nil
 }
+
+// ConvertToByItemExpr rewrites ByItem.ExprNode to a proper ExprNode.
+func ConvertToByItemExpr(n ast.Node) ast.Node {


how about:

func ConstructPositionExpr(v *driver.ParamMarkerExpr) *ast.PositionExpr { return &ast.PositionExpr{P: v} }

In order to do that, we must add the checking logic of driver.ParamMarkerExpr typed node before calling ConstructPositionExpr. Hence, IMHO, the current impl. might be simpler than your suggestions.

[jianzhang.zj:~/opt/gopath/src/github.com/pingcap/tidb] git:(cloud-pi/fix-the-issue-8153 ✔) ➜ ag -w ConvertToByItemExpr planner/core/logical_plan_builder.go 827: newNode := expression.ConvertToByItemExpr(inNode) 1207: newNode := expression.ConvertToByItemExpr(inNode) expression/util.go 531:// ConvertToByItemExpr rewrites ByItem.ExprNode to a proper ExprNode. 532:func ConvertToByItemExpr(n ast.Node) ast.Node {

These are all the appearance of ConvertToByItemExpr, and as you can see in the function caller, the input parameter is guaranteed to be driver.ParamMarkerExpr

…ache is enabled

zz-jason

LGTM

…cap#8206)

dbjoa mentioned this pull request Nov 6, 2018

parser,ast: fix the TiDB issue #8153 pingcap/parser#20

Merged

zz-jason reviewed Nov 6, 2018

View reviewed changes

planner/core/expression_rewriter.go Outdated Show resolved Hide resolved

zz-jason added contribution This PR is from a community contributor. type/bugfix This PR fixes a bug. sig/planner SIG: Planner labels Nov 6, 2018

eurekaka reviewed Nov 6, 2018

View reviewed changes

planner/core/logical_plan_builder.go Show resolved Hide resolved

planner/core/expression_rewriter.go Outdated Show resolved Hide resolved

planner/core/expression_rewriter.go Outdated Show resolved Hide resolved

dbjoa changed the title ~~executor,planner: fix the issue #8153~~ [WIP] executor,planner: fix the issue #8153 Nov 6, 2018

dbjoa mentioned this pull request Nov 8, 2018

ast: fix the TiDB issue #8153 pingcap/parser#27

Merged

dbjoa changed the title ~~[WIP] executor,planner: fix the issue #8153~~ executor,planner: fix the issue #8153 Nov 8, 2018

shenli added the status/all tests passed label Nov 8, 2018

XuHuaiyu changed the title ~~executor,planner: fix the issue #8153~~ executor,planner: make order by ? performs correctly when prepare-cache is enabled Nov 12, 2018

winoros reviewed Nov 15, 2018

View reviewed changes

planner/core/logical_plan_builder.go Outdated Show resolved Hide resolved

planner/core/logical_plan_builder.go Outdated Show resolved Hide resolved

dbjoa mentioned this pull request Nov 30, 2018

[experimental] plan: set the default value of ParamMarkerExpr to NULL #8536

Closed

eurekaka reviewed Dec 3, 2018

View reviewed changes

eurekaka added the status/LGT1 Indicates that a PR has LGTM 1. label Dec 3, 2018

eurekaka changed the title ~~executor,planner: make order by ? performs correctly when prepare-cache is enabled~~ plan: support Order By ? | Group By ? | Limit ? Offset ? Dec 3, 2018

eurekaka changed the title ~~plan: support Order By ? | Group By ? | Limit ? Offset ?~~ plan: support ParamMarkerExpr in Order By / Group By / Limit Offset clauses Dec 3, 2018

eurekaka changed the title ~~plan: support ParamMarkerExpr in Order By / Group By / Limit Offset clauses~~ plan: support ? in Order By / Group By / Limit Offset clauses Dec 3, 2018

winoros reviewed Dec 3, 2018

View reviewed changes

expression/util.go Show resolved Hide resolved

winoros reviewed Dec 3, 2018

View reviewed changes

zz-jason reviewed Dec 3, 2018

View reviewed changes

dbjoa added 11 commits December 3, 2018 22:25

executor,planner: make order by ? performs correctly when prepare-c…

c963458

…ache is enabled

address the comments

c42c2b2

fix a potential sporadic CI error due to the PR #8339

7ff1e1c

correct the condition

2c1e00c

code clean up

5716434

cleaning the codes

07ae3ca

fix a potential error

14201c7

fix error messages

3f78986

remove StmtCtx.InPrepare and set the default parameter value as NULL

105d4f4

remove the unnecessary trace

253106d

clean up the codes

7ac5db0

zz-jason approved these changes Dec 3, 2018

View reviewed changes

zz-jason added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Dec 3, 2018

zz-jason merged commit c677187 into pingcap:master Dec 3, 2018

iamzhoug37 pushed a commit to iamzhoug37/tidb that referenced this pull request Dec 13, 2018

plan: support ? in Order By / Group By / Limit Offset clauses (ping…

fbd8dd1

…cap#8206)

zz-jason pushed a commit to zz-jason/tidb that referenced this pull request Oct 3, 2019

plan: support ? in Order By / Group By / Limit Offset clauses (ping…

c0c1360

…cap#8206)

zz-jason mentioned this pull request Oct 3, 2019

plan: support ? in Order By / Group By / Limit Offset clauses (#8206) #12514

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

plan: support `?` in Order By / Group By / Limit Offset clauses #8206

plan: support `?` in Order By / Group By / Limit Offset clauses #8206

dbjoa commented Nov 6, 2018 •

edited by c4pt0r

Loading

sre-bot commented Nov 6, 2018

zz-jason commented Nov 6, 2018

zz-jason commented Nov 6, 2018

dbjoa commented Nov 6, 2018

dbjoa commented Nov 6, 2018

zz-jason commented Nov 6, 2018

tiancaiamao commented Nov 7, 2018

dbjoa commented Nov 7, 2018

dbjoa commented Nov 7, 2018

eurekaka commented Nov 8, 2018

dbjoa commented Nov 8, 2018

dbjoa commented Nov 8, 2018 •

edited

Loading

shenli commented Nov 8, 2018

zz-jason commented Nov 15, 2018

dbjoa commented Nov 15, 2018

dbjoa commented Nov 15, 2018

dbjoa commented Nov 15, 2018

dbjoa commented Nov 15, 2018

dbjoa commented Nov 30, 2018 •

edited

Loading

eurekaka left a comment

winoros Dec 3, 2018

dbjoa Dec 3, 2018

crazycs520 commented Dec 3, 2018

dbjoa commented Dec 3, 2018

zz-jason Dec 3, 2018

dbjoa Dec 3, 2018 •

edited

Loading

zz-jason Dec 3, 2018

dbjoa Dec 3, 2018

zz-jason left a comment

plan: support ? in Order By / Group By / Limit Offset clauses #8206

plan: support ? in Order By / Group By / Limit Offset clauses #8206

Conversation

dbjoa commented Nov 6, 2018 • edited by c4pt0r Loading

What problem does this PR solve?

What is changed and how it works?

Check List

sre-bot commented Nov 6, 2018

zz-jason commented Nov 6, 2018

zz-jason commented Nov 6, 2018

dbjoa commented Nov 6, 2018

dbjoa commented Nov 6, 2018

zz-jason commented Nov 6, 2018

tiancaiamao commented Nov 7, 2018

dbjoa commented Nov 7, 2018

dbjoa commented Nov 7, 2018

eurekaka commented Nov 8, 2018

dbjoa commented Nov 8, 2018

dbjoa commented Nov 8, 2018 • edited Loading

shenli commented Nov 8, 2018

zz-jason commented Nov 15, 2018

dbjoa commented Nov 15, 2018

dbjoa commented Nov 15, 2018

dbjoa commented Nov 15, 2018

dbjoa commented Nov 15, 2018

dbjoa commented Nov 30, 2018 • edited Loading

eurekaka left a comment

Choose a reason for hiding this comment

winoros Dec 3, 2018

Choose a reason for hiding this comment

dbjoa Dec 3, 2018

Choose a reason for hiding this comment

crazycs520 commented Dec 3, 2018

dbjoa commented Dec 3, 2018

zz-jason Dec 3, 2018

Choose a reason for hiding this comment

dbjoa Dec 3, 2018 • edited Loading

Choose a reason for hiding this comment

zz-jason Dec 3, 2018

Choose a reason for hiding this comment

dbjoa Dec 3, 2018

Choose a reason for hiding this comment

zz-jason left a comment

Choose a reason for hiding this comment

plan: support `?` in Order By / Group By / Limit Offset clauses #8206

plan: support `?` in Order By / Group By / Limit Offset clauses #8206

dbjoa commented Nov 6, 2018 •

edited by c4pt0r

Loading

dbjoa commented Nov 8, 2018 •

edited

Loading

dbjoa commented Nov 30, 2018 •

edited

Loading

dbjoa Dec 3, 2018 •

edited

Loading