Update AST helper generation #7558

systay · 2021-02-26T15:02:10Z

Description

This PR removes the visitorgen component that used to create the Vitess Rewriter, and adds asthelpergen that generates both the rewriter, but also deep-clone methods for all AST types.

The Gen4 planner tries join in both directions. Given the following query:

SELECT a.col, b.col FROM A JOIN B ON a.id = b.id

The new planner will try to first plan the A side on the left, and the send data from A to B as params, and then will try the other variation. The two versions would be:

-- ********** A JOIN B
-- A side
SELECT a.col FROM A
-- B side
SELECT b.col FROM B WHERE ? = b.id



-- ********** B JOIN A
-- B side
SELECT b.col FROM B
-- B side
SELECT b.col FROM A WHERE a.id = ?

As can be seen in these examples, we need to be able to clone the original predicate and then change it in two different ways to test both alternatives.

Related Issue(s)

#7280

Checklist

Should this PR be backported? - Absolutely not
Tests were added or are not required - Lots of tests added
Documentation was added or is not required - No docs needed. Internal change only

Impacted Areas in Vitess

Components that this PR will affect:

Query Serving

Signed-off-by: Andres Taylor <andres@planetscale.com>

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

Signed-off-by: Andres Taylor <andres@planetscale.com>

Signed-off-by: GuptaManan100 <manan@planetscale.com>

Signed-off-by: Andres Taylor <andres@planetscale.com>

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

vmg

Looks good to me! We spent quite a bit of time reviewing this together. ✨

GuptaManan100 · 2021-03-01T16:52:02Z

go/tools/asthelpergen/asthelpergen.go

+	return nil
+}
+
+func (gen *astHelperGen) GenerateCode() (map[string]*jen.File, error) {


Do we need a comment for an exported function?

Signed-off-by: Andres Taylor <andres@planetscale.com>

This reverts commit cd7c4b3. Turns out this change significantly degrades performance, and we don't want that... Signed-off-by: Andres Taylor <andres@planetscale.com>

Signed-off-by: Andres Taylor <andres@planetscale.com>

vmg · 2021-03-02T11:14:41Z

@systay I now see where the performance regression in cd7c4b3 comes from -- you're actually allocating closures now! By removing the extra argument to every replacement function, all the visitor callbacks that were previously pure functions are becoming closures, which require an allocation even if they're not being called. Definitely a performance regression.

harshit-gangal

Really nice work.

harshit-gangal · 2021-03-02T11:53:18Z

go/tools/asthelpergen/integration/types.go

+const (
+	// these consts are here to try to trick the generator
+	thisIsNotAType  BasicType = 1
+	thisIsNotAType2 BasicType = 2
+)


looks like they are not used in the code.

the comment is not lying. these consts produce code that we had to do extra checks in the generator code to avoid. They are here to mimic sqlparser types such as ReadOnly (https://github.com/vitessio/vitess/blob/master/go/vt/sqlparser/constants.go#L254)

harshit-gangal · 2021-03-02T11:54:59Z

go/tools/asthelpergen/integration/integration_rewriter_test.go

+func (tv *testVisitor) assertEquals(t *testing.T, expected []step) {
+	t.Helper()
+	var lines []string
+	error := false


nit: error is reserved word, better to name as err

systay and others added 30 commits February 12, 2021 07:49

it's a start

71482a7

Signed-off-by: Andres Taylor <andres@planetscale.com>

apply pre- and post-switch

4cbdb20

Signed-off-by: Andres Taylor <andres@planetscale.com>

switch

08e38d9

Signed-off-by: Andres Taylor <andres@planetscale.com>

extracted rewriter code to own file

0a1992e

Signed-off-by: Andres Taylor <andres@planetscale.com>

visit struct fields that implement the interface

05b0af3

Signed-off-by: Andres Taylor <andres@planetscale.com>

support field with struct type

d72873d

Signed-off-by: Andres Taylor <andres@planetscale.com>

extracted rewriter generation into its own file

9290991

Signed-off-by: Andres Taylor <andres@planetscale.com>

added replacer methods

9c3dc09

Signed-off-by: Andres Taylor <andres@planetscale.com>

support interface implementations that are not pointers

e2d23e2

Signed-off-by: Andres Taylor <andres@planetscale.com>

add replacement methods for array fields

98e4e5b

Signed-off-by: Andres Taylor <andres@planetscale.com>

added case statements for slice fields

5c93919

Signed-off-by: Andres Taylor <andres@planetscale.com>

Addition of panic of value type implementation

2d66e2e

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

asthelpergen: improve integration tests

999a36f

Signed-off-by: Andres Taylor <andres@planetscale.com>

better tests

21e902c

Signed-off-by: Andres Taylor <andres@planetscale.com>

benchmark for slice replacer

7af230c

Signed-off-by: Andres Taylor <andres@planetscale.com>

nicer replacement methods for slices

7b5f171

Signed-off-by: Andres Taylor <andres@planetscale.com>

Added replacer and cases for InterfaceSlice

0116a12

Signed-off-by: GuptaManan100 <manan@planetscale.com>

moved code away from the generated file

c3c68be

Signed-off-by: Andres Taylor <andres@planetscale.com>

support slices of non-AST elements

0fb0313

Signed-off-by: Andres Taylor <andres@planetscale.com>

use the new rewriter

3f43634

Signed-off-by: Andres Taylor <andres@planetscale.com>

Merge remote-tracking branch 'upstream/master' into visitorgen-rewrite3

34c6502

Signed-off-by: Andres Taylor <andres@planetscale.com>

refactored and addressed review comments

f2d4e68

Signed-off-by: Andres Taylor <andres@planetscale.com>

Merge remote-tracking branch 'upstream/master' into visitorgen-rewrite3

99b6ac0

Signed-off-by: Andres Taylor <andres@planetscale.com>

Add deep clone capability to ASThelpergen

ffe3527

Signed-off-by: Andres Taylor <andres@planetscale.com>

added support for basic types that implement interfaces

725fcd5

Signed-off-by: Andres Taylor <andres@planetscale.com>

handle interface{} fields

e2dc629

Signed-off-by: Andres Taylor <andres@planetscale.com>

final touch ups. produce the clone.go file for the AST

044b550

Signed-off-by: Andres Taylor <andres@planetscale.com>

code clean up; minor change for interface clone methods

b42085d

Signed-off-by: Andres Taylor <andres@planetscale.com>

make the clone methods use less memory

b79366a

Signed-off-by: Andres Taylor <andres@planetscale.com>

added comment to clone methods

858859e

Signed-off-by: Andres Taylor <andres@planetscale.com>

remove the hand written clone and start using the generated one

dc14a65

Signed-off-by: Andres Taylor <andres@planetscale.com>

systay changed the title ~~Deep clone~~ Update AST helper generation Feb 27, 2021

systay marked this pull request as ready for review February 27, 2021 14:23

systay requested review from GuptaManan100, harshit-gangal and sougou as code owners February 27, 2021 14:23

inline replacer methods for the rewriter

09f3f0b

Signed-off-by: Andres Taylor <andres@planetscale.com>

This was referenced Feb 27, 2021

Visitorgen rewrite #7534

Closed

Gen4 Tracking #7280

Closed

systay and others added 2 commits February 27, 2021 17:16

Merge remote-tracking branch 'upstream/master' into deep-clone

fce3959

Signed-off-by: Andres Taylor <andres@planetscale.com>

made nextval as reference type in ast

aa88caf

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

systay requested a review from shlomi-noach as a code owner March 1, 2021 06:57

vmg approved these changes Mar 1, 2021

View reviewed changes

GuptaManan100 approved these changes Mar 1, 2021

View reviewed changes

systay added 7 commits March 2, 2021 09:11

dry

3e95345

Signed-off-by: Andres Taylor <andres@planetscale.com>

simplify the function used to visit slice elements

f4036ca

Signed-off-by: Andres Taylor <andres@planetscale.com>

simplify the rewriter some more

cd7c4b3

Signed-off-by: Andres Taylor <andres@planetscale.com>

rewriter benchmark

f03d4b1

Signed-off-by: Andres Taylor <andres@planetscale.com>

Revert "simplify the rewriter some more"

9af91e0

This reverts commit cd7c4b3. Turns out this change significantly degrades performance, and we don't want that... Signed-off-by: Andres Taylor <andres@planetscale.com>

added error to the return params of Rewrite

5488cc6

Signed-off-by: Andres Taylor <andres@planetscale.com>

return errors instead of panicking

db053f7

Signed-off-by: Andres Taylor <andres@planetscale.com>

systay requested review from deepthi and rohit-nayak-ps as code owners March 2, 2021 10:28

harshit-gangal approved these changes Mar 2, 2021

View reviewed changes

harshit-gangal merged commit a75c159 into vitessio:master Mar 2, 2021

harshit-gangal deleted the deep-clone branch March 2, 2021 12:30

askdba added the Component: Query Serving label Mar 4, 2021

askdba added this to the v10.0 milestone Mar 4, 2021

ajm188 mentioned this pull request Jul 14, 2021

slack vitess v10.pre tinyspeck/vitess#228

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update AST helper generation #7558

Update AST helper generation #7558

systay commented Feb 26, 2021 •

edited by arvind-murty

Loading

vmg left a comment

GuptaManan100 Mar 1, 2021

vmg commented Mar 2, 2021

harshit-gangal left a comment

harshit-gangal Mar 2, 2021

systay Mar 2, 2021

harshit-gangal Mar 2, 2021

Update AST helper generation #7558

Update AST helper generation #7558

Conversation

systay commented Feb 26, 2021 • edited by arvind-murty Loading

Description

Related Issue(s)

Checklist

Impacted Areas in Vitess

vmg left a comment

Choose a reason for hiding this comment

GuptaManan100 Mar 1, 2021

Choose a reason for hiding this comment

vmg commented Mar 2, 2021

harshit-gangal left a comment

Choose a reason for hiding this comment

harshit-gangal Mar 2, 2021

Choose a reason for hiding this comment

systay Mar 2, 2021

Choose a reason for hiding this comment

harshit-gangal Mar 2, 2021

Choose a reason for hiding this comment

systay commented Feb 26, 2021 •

edited by arvind-murty

Loading