Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Backport 14.0] enable schema tracking by default #10595

Merged
merged 5 commits into from
Jun 28, 2022
Merged

[Backport 14.0] enable schema tracking by default #10595

merged 5 commits into from
Jun 28, 2022

Conversation

harshit-gangal
Copy link
Member

Description

Backport of #10455

This PR enables schema tracking by default. It's mostly about changing default settings and then updating integration tests that start failing.

Related Issue(s)

Checklist

  • Tests were added or are not required
  • Documentation was added or is not required

Deployment Notes

@harshit-gangal harshit-gangal added Type: Enhancement Logical improvement (somewhere between a bug and feature) Component: Query Serving Backport This is a backport release notes NeedsWebsiteDocsUpdate What it says labels Jun 28, 2022
@vitess-bot
Copy link
Contributor

vitess-bot bot commented Jun 28, 2022 •
edited by frouioui
Loading

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

  • Ensure that the Pull Request has a descriptive title.
  • If this is a change that users need to know about, please apply the release notes (needs details) label so that merging is blocked unless the summary release notes document is included.
  • If a new flag is being introduced, review whether it is really needed. The flag names should be clear and intuitive (as far as possible), and the flag's help should be descriptive.
  • If a workflow is added or modified, each items in Jobs should be named in order to mark it as required. If the workflow should be required, the GitHub Admin should be notified.

Bug fixes

  • There should be at least one unit or end-to-end test.
  • The Pull Request description should either include a link to an issue that describes the bug OR an actual description of the bug and how to reproduce, along with a description of the fix.

Non-trivial changes

  • There should be some code comments as to why things are implemented the way they are.

New/Existing features

  • Should be documented, either by modifying the existing documentation or creating new documentation.
  • New features should have a link to a feature request issue or an RFC that documents the use cases, corner cases and test cases.

Backward compatibility

  • Protobuf changes should be wire-compatible.
  • Changes to _vt tables and RPCs need to be backward compatible.
  • vtctl command output order should be stable and awk-able.

@harshit-gangal harshit-gangal changed the base branch from main to release-14.0 June 28, 2022 07:26
@harshit-gangal @systay @rohit-nayak-ps
enable schema tracking by default (vitessio#10455)
3d5f18a 
* feat: enable schema tracking by default

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

* test: fix test setup

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

* test: fix vschema test setup

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

* test: turn off schema tracking on the tablet

Signed-off-by: Andres Taylor <andres@planetscale.com>

* test: fix test assertion

Signed-off-by: Andres Taylor <andres@planetscale.com>

* Change read query for checks that test to which keyspace a table is routed to. This uses the /queryz vttablet endpoint where the query gets expanded if schema tracking is enabled, hence failing an exact query check

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

* test: only use gen4 planner

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

* fix: column list population in insert with auto-inc column

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

* fix: change parser to keep empty column list as provided by user

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

* test: fix test expectation

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

Co-authored-by: Andres Taylor <andres@planetscale.com>
Co-authored-by: Rohit Nayak <rohit@planetscale.com>
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
@harshit-gangal harshit-gangal changed the title Bp 10455 [Backport 14.0] enable schema tracking by default Jun 28, 2022
harshit-gangal and others added 2 commits June 28, 2022 13:40
@harshit-gangal
test: fix test expectation
dcde88a 
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
@systay
fix: Allow gen4 to handle variables on authoritative tables
3cfcde1 
Signed-off-by: Andres Taylor <andres@planetscale.com>
@systay systay mentioned this pull request Jun 28, 2022
Closed
3 tasks
@harshit-gangal
Empty-Commit
c834e21 
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
@harshit-gangal
test: system variable selection on a table only works in gen4
bb29b63 
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
@frouioui frouioui mentioned this pull request Jun 28, 2022
Closed
43 tasks
@GuptaManan100 GuptaManan100 merged commit 8ebe1cc into vitessio:release-14.0 Jun 28, 2022
@GuptaManan100 GuptaManan100 deleted the bp-10455 branch June 28, 2022 12:39
@deepthi deepthi mentioned this pull request Jun 28, 2022
Merged
3 tasks
systay added a commit to planetscale/vitess that referenced this pull request Aug 19, 2022
@vmg @rsajwani
@systay @K-Kumar-01 @GuptaManan100 @frouioui @harshit-gangal @rohit-nayak-ps @dbussink
Cherry Pick all SQLParser changes (vitessio#904)
c07c82b 
* Refactor aggregation AST structs (vitessio#10347)

* SQLParser:Refactoring Add count struct

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

* SQLParser:Refactoring Add countStar struct

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

* SQLParser:Refactoring Add avg struct

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

* SQLParser:Refactoring Add max struct

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

* SQLParser:Refactoring Add min struct

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

* SQLParser:Refactoring Add sum struct

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

* SQLParser:Refactoring Fixing Parser Aggr Function

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

* fix: the return type of count was wrong

Signed-off-by: Andres Taylor <andres@planetscale.com>

* Refacotring code

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

* More refactoring and unit test cases fix

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

* removing getarg from aggregate interface

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

* Fixing bugs after merge

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

* Optimizing code

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

* Fixing planBuilder test cases`

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

* Adding more aggregate functions

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

* Fixing parser errors

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

* fixing feedback , code review

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

* Fix regression which making changes in previous commit

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

* Fixing vdiff tests

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

* Fixing replication test

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

* Fixing vstreamer planbuilder test

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

* remove redundant code

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

Co-authored-by: Andres Taylor <andres@planetscale.com>
Signed-off-by: Vicent Marti <vmg@strn.cat>

* Add parsing support for performance schema functions (vitessio#10478)

* feat: add parsing support for performance schema functions

Signed-off-by: Kushal Kumar <kushalkumargupta4@gmail.com>

* fix: update sql.go by making parser

Signed-off-by: Kushal Kumar <kushalkumargupta4@gmail.com>
Signed-off-by: Vicent Marti <vmg@strn.cat>

* Reduce shift-reduce conflicts (vitessio#10500)

* feat: reduce shift-reduce conflicts by using the precedence symbol FUNCTION_CALL_NON_KEYWORD

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: fix static check workflow to setup go in cases of parser changes too

Signed-off-by: Manan Gupta <manan@planetscale.com>
Signed-off-by: Vicent Marti <vmg@strn.cat>

* Cleanup:  Remove 'Name' field from aggregate structure (vitessio#10507)

* Code clean up

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

* Fixing unit test case failures

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

* Fixing replicator planner unit tests

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

* Adding unit test cases of canonical output

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

* Fix unit test

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>
Signed-off-by: Vicent Marti <vmg@strn.cat>

* User defined and sys variables (vitessio#10547)

* feat: add user defined variables and sys variables as separate structs and use user-defined variable in Execute statement

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: remove parsing of at_id and at_at_id from places that shouldn't parse it

Signed-off-by: Manan Gupta <manan@planetscale.com>
Signed-off-by: Andres Taylor <andres@planetscale.com>

* feat: use ci_identifier in more places

Signed-off-by: Andres Taylor <andres@planetscale.com>

* refactor: clean up the JSON AST structs a little

Signed-off-by: Andres Taylor <andres@planetscale.com>

* refactor: use remove unnesseccary scope

Signed-off-by: Andres Taylor <andres@planetscale.com>

* Created new Variable node to divide the work of ColName

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

* refactor fix sqlparser: SET ast and parsing

Signed-off-by: Andres Taylor <andres@planetscale.com>

* refactor: remove unused ast-structs

Signed-off-by: Andres Taylor <andres@planetscale.com>

* refactor: clean up how variables are parsed and formatted

Signed-off-by: Andres Taylor <andres@planetscale.com>

* fix: make the new AST work in the planbuilder

Signed-off-by: Andres Taylor <andres@planetscale.com>

* refactor: rename ColIdent and TableIdent

Signed-off-by: Andres Taylor <andres@planetscale.com>

* fix: set name should be parsed with SessionScope

Signed-off-by: Andres Taylor <andres@planetscale.com>

* chore: clean and document stuff

Signed-off-by: Andres Taylor <andres@planetscale.com>

* test: update test expectation

Signed-off-by: Andres Taylor <andres@planetscale.com>

* fix: default scope for transaction isolation should not be session

Signed-off-by: Andres Taylor <andres@planetscale.com>

* feat: handle local as a synonym for session for variable scopes

Signed-off-by: Andres Taylor <andres@planetscale.com>

* comments: clean up code comments

Signed-off-by: Andres Taylor <andres@planetscale.com>

Co-authored-by: Manan Gupta <manan@planetscale.com>
Co-authored-by: Florent Poinsard <florent.poinsard@outlook.fr>
Signed-off-by: Vicent Marti <vmg@strn.cat>

* Revert "[Backport 14.0] enable schema tracking by default  (vitessio#10595)"

This reverts commit 8ebe1cc.

Signed-off-by: Vicent Marti <vmg@strn.cat>

* enable schema tracking by default (vitessio#10455)

* feat: enable schema tracking by default

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

* test: fix test setup

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

* test: fix vschema test setup

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

* test: turn off schema tracking on the tablet

Signed-off-by: Andres Taylor <andres@planetscale.com>

* test: fix test assertion

Signed-off-by: Andres Taylor <andres@planetscale.com>

* Change read query for checks that test to which keyspace a table is routed to. This uses the /queryz vttablet endpoint where the query gets expanded if schema tracking is enabled, hence failing an exact query check

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

* test: only use gen4 planner

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

* fix: column list population in insert with auto-inc column

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

* fix: change parser to keep empty column list as provided by user

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

* test: fix test expectation

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

Co-authored-by: Andres Taylor <andres@planetscale.com>
Co-authored-by: Rohit Nayak <rohit@planetscale.com>
Signed-off-by: Vicent Marti <vmg@strn.cat>

* Add parsing support for GTID functions (vitessio#10579)

* feat: add parsing support for GTID functions

Signed-off-by: Kushal Kumar <kushalkumargupta4@gmail.com>

* test: add planner and end to end tests for a few gtid functions

Signed-off-by: Manan Gupta <manan@planetscale.com>

Co-authored-by: Manan Gupta <manan@planetscale.com>
Signed-off-by: Vicent Marti <vmg@strn.cat>

* Parameterize BIT types and fixes in HEX types (vitessio#10689)

* feat: add parsing for bitnums

Signed-off-by: Manan Gupta <manan@planetscale.com>

* test: add invalid cases to tests

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: normalize bitnums to bit vals too

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: add normalization for bit literals

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: parameterize binary value to hex

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

* test: added e2e test in vtgate and vttablet

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

* fix: fix the type conversion from hexnum and hexval to binary

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

* test: refactor test

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

* fix: return varbinary and flaghex for hexval and hexnum in eval engine typeOf

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

* test: separate expectation for mysql and vitess result

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

* test: fix test expectation

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

* feat: added bitnum bind variable and test

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

* proto: vtadmin side update

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

* test: fixed test and added more bitnum bind var test

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

* added license header

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

Co-authored-by: Manan Gupta <manan@planetscale.com>
Signed-off-by: Vicent Marti <vmg@strn.cat>

* Mark aggregate functions callable (vitessio#10805)

These should also implement the iCallable interface as they are also
callable as functions.

Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
Signed-off-by: Vicent Marti <vmg@strn.cat>

* cherry-pick: fix bad merge

Signed-off-by: Vicent Marti <vmg@strn.cat>

Co-authored-by: rsajwani <rameezwazirali@hotmail.com>
Co-authored-by: Andres Taylor <andres@planetscale.com>
Co-authored-by: Kushal Kumar <59891164+K-Kumar-01@users.noreply.github.com>
Co-authored-by: Manan Gupta <35839558+GuptaManan100@users.noreply.github.com>
Co-authored-by: Manan Gupta <manan@planetscale.com>
Co-authored-by: Florent Poinsard <florent.poinsard@outlook.fr>
Co-authored-by: Harshit Gangal <harshit@planetscale.com>
Co-authored-by: Rohit Nayak <rohit@planetscale.com>
Co-authored-by: Dirkjan Bussink <d.bussink@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@systay systay systay approved these changes

@frouioui frouioui frouioui approved these changes

@deepthi deepthi Awaiting requested review from deepthi deepthi is a code owner

@rohit-nayak-ps rohit-nayak-ps Awaiting requested review from rohit-nayak-ps rohit-nayak-ps is a code owner

@GuptaManan100 GuptaManan100 Awaiting requested review from GuptaManan100 GuptaManan100 is a code owner

Assignees
No one assigned
Labels
Backport This is a backport Component: Query Serving NeedsWebsiteDocsUpdate What it says Type: Enhancement Logical improvement (somewhere between a bug and feature)
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@harshit-gangal @systay @frouioui @GuptaManan100