declarative Online DDL

[shlomi-noach]

Description

This PR introduces (partial) declarative schema changes in Online DDL. In a declarative schema change you tell vitess "this is a table's schema, make it so". If the table does not exist, vitess creates it. If the table exists and with a different schema, vitess evaluates the required ALTER statement and migrates the table.

Declarative changes have the property of being idempotent. You may issue the same migration multiple times with no harm done. If vitess find that no change is required, it implicitly markes the migration as successful.

This PR implements partial declarative changes. In a full declarative schema change, one gives their database a list of all tables in a schema. In a partial declarative schema change, we give vitess a single table at a time.

This means:

• If you want to create a table, ask vitess to CREATE TABLE
• If you want to modify a table, ask vitess to CREATE TABLE
• If you want to drop a table, ask vitess to DROP TABLE.

Vitess supports the -declarative flag in dddl_strategy to enable this functionality.

Usage

consider:

set @@ddl_strategy='online -declarative';
create table t(id int primary key); -- table created
create table t(id int primary key, t text); -- table modified
create table t(id int primary key, t text); -- no changes, implicitly successful
create table t(id int primary key, t text, key t_idx(t(32))); -- table modified
drop table t; -- table dropped
drop table t; -- no changes, implicitly successful

you may use -declarative in all online DDL strategies: online, gh-ost, pt-osc.

Documentation to follow.

Related Issue(s)

Tracking: #6926

Checklist

• Should this PR be backported?
• Tests were added or are not required
• Documentation was added or is not required

Deployment Notes

Impacted Areas in Vitess

Components that this PR will affect:

• Query Serving
• VReplication
• Cluster Management
• Build/CI
• VTAdmin

[shlomi-noach]

Reviewers: the main changes to review are in executor.go:

• new evaluateDeclarativeDiff() function
• and, within executeMigration() function

when we handle a new migration request, we now first check to see if it's declarative. If so, we may end up actually running a modified migration than defined. We break down into the following cases:

• it's a revert (no special behavior)
• alter is not allowed
• drop: silently mark as successful if the table does not exist. Otherwise treat that as a normal drop
• create: either
  • the table does not exist -> proceed as normal create
  • the table exists -> use tengo to evaluate the diff statement from existing table into desired table. Either:
    • there is no diff (exact same schema): silently mark as successful
    • there is a diff: modify into an alter statement, to be handled later on.

Diffing the table via tengo:

• tengo is a library which uses sqlx to connect to to the database
• we have slightly modified tengo, https://github.com/planetscale/tengo
• we create an account and connection for sqlx & tengo
• we create a temporary table within the tablet server's schema (keyspace), which has the new design
• we diff a the specific original table with this temporary table
  • tengo doesn't like diffing tables of different names, because it assumes identical names in different schemas. We hack around by rewriting tengo's analysis of the table
• the diff is a SQL string, which is the ALTER options (e.g add column i int, drop key k), or empty if the tables are identical

[mavenraven]

@shlomi-noach Awesome! I'm not going to review the implementation unless you'd like me to. From a perspective of a user of the functionality, it looks perfect.

[rohit-nayak-ps]

lgtm. This is really nice!

[rohit-nayak-ps]

th eappropriate handlers => the appropriate handlers

[shlomi-noach]

fixed

[rohit-nayak-ps]

should be DROP here and not CREATE?

[shlomi-noach]

fixed!