Vitess v14.0.1

Release of Vitess v14.0.1

Known Issues

• Corrupted results for non-full-group-by queries with JOINs. This can be resolved by using full-group-by queries.

Major Changes

Upgrade to go1.18.4

Vitess v14.0.1 now runs on go1.18.4.
The patch release of Go, go1.18.4, was one of main motivations for this release as it includes important security fixes to packages used by Vitess.
Below is a summary of this patch release. You can learn more here.

go1.18.4 (released 2022-07-12) includes security fixes to the compress/gzip, encoding/gob, encoding/xml, go/parser, io/fs, net/http, and path/filepath packages, as well as bug fixes to the compiler, the go command, the linker, the runtime, and the runtime/metrics package. See the Go 1.18.4 milestone on our issue tracker for details.

---

The entire changelog for this release can be found here.

The release includes 25 commits (excluding merges)

Thanks to all our contributors: @GuptaManan100, @deepthi, @frouioui, @harshit-gangal, @mattlord, @rohit-nayak-ps, @shlomi-noach, @vitess-bot[bot]

Vitess v13.0.2

Release of Vitess v13.0.2

Major Changes

Upgrade to go1.17.12

Vitess v13.0.2 now runs on go1.17.12.
The patch release of Go, go1.17.12, was one of main motivations for this release as it includes important security fixes to packages used by Vitess.
Below is a summary of this patch release. You can learn more here.

go1.17.12 (released 2022-07-12) includes security fixes to the compress/gzip, encoding/gob, encoding/xml, go/parser, io/fs, net/http, and path/filepath packages, as well as bug fixes to the compiler, the go command, the runtime, and the runtime/metrics package. See the Go 1.17.12 milestone on our issue tracker for details.

---

The entire changelog for this release can be found here.
The release includes 28 commits (excluding merges)

Thanks to all our contributors: @GuptaManan100, @aquarapid, @frouioui, @harshit-gangal, @mattlord, @rohit-nayak-ps, @systay, @vitess-bot[bot], @vmg

Vitess v14.0.0

Release of Vitess v14.0.0

Summary

• Gen4 is now the default planner
• New query support
• Command-line syntax deprecations
• New command line flags and behavior
• Online DDL changes
• Table lifecycle
• Tablet throttler
• Heartbeat
• VDiff2
• Durability Policy
• Deprecation of Durability Configuration
• Advisory locking optimizations
• Pre-Legacy Resharding is now deprecated

Known Issues

• VTOrc doesn't discover the tablets of a keyspace if the durability policy doesn't exist in the topo server when it comes up. This can be resolved by restarting VTOrc.
• Corrupted results for non-full-group-by queries with JOINs. This can be resolved by using full-group-by queries.

Major Changes

Gen4 is now the default planner

The new planner has been in the works since end of 2020, and it's finally grown enough to be able to become the default planner for Vitess.
This means that many more queries are supported on sharded keyspaces, and old queries might get planned better than before.
You can always roll back to the earlier planner, either by providing the flag --planner-version=V3 to vtgate, or by adding a comment to individual queries, like so:

select /*vt+ PLANNER=V3 */ name, count(*) from users

New query support

Support for aggregation across shards

Vitess can now plan and execute most aggregation queries across multiple shards and/or keyspaces.

INSERT from SELECT

Support has been added for inserting new data from SELECT queries.
Now you can insert data from a query into a table using a query like:

insert into tbl (col) select id from users 

UPDATE from SELECT

Similarly, we have added support for UPDATE with scalar sub-queries. This allows for queries where the updated value is fetched using a subquery, such as this example:

update tbl set foo = (select count(*) from otherTbl)

Command-line syntax deprecations

Vitess has begun a transition to a new library for CLI flag parsing.
In order to facilitate a smooth transition, certain syntaxes that will not be supported in the future now issue deprecation warnings when used.

The messages you will likely see, along with explanations and migrations, are:

"Use of single-dash long flags is deprecated"

Single-dash usage will be only possible for short flags (e.g. -v is okay, but -verbose is not).

To migrate, update your CLI scripts from:

$ vttablet -tablet_alias zone1-100 -init_keyspace mykeyspace ... # old way

To:

$ vttablet --tablet_alias zone1-100 --init_keyspace mykeyspace ... # new way

"Detected a dashed argument after a position argument."

As the full deprecation text goes on to (attempt to) explain, mixing flags and positional arguments will change in a future version that will break scripts.

Currently, when invoking a binary like:

$ vtctl --topo_implementation etcd2 AddCellInfo --root "/vitess/global"

Everything after the AddCellInfo is treated by package flag as a positional argument, and we then use a sub FlagSet to parse flags specific to the subcommand.
So, at the top-level, flag.Args() returns ["AddCellInfo", "--root", "/vitess/global"].

The library we are transitioning to is more flexible, allowing flags and positional arguments to be interwoven on the command-line.
For the above example, this means that we would attempt to parse --root as a top-level flag for the VTCtl binary.
This will cause the program to exit on error, because that flag is only defined on the AddCellInfo subcommand.

In order to transition, a standalone double-dash (literally, --) will cause the new flag library to treat everything following that as a positional argument, and also works with the current flag parsing code we use.

So, to transition the above example without breakage, update the command to:

$ vtctl --topo_implementation etcd2 AddCellInfo -- --root "/vitess/global"
$ # the following will also work
$ vtctl --topo_implementation etcd2 -- AddCellInfo --root "/vitess/global"
$ # the following will NOT work, because --topo_implementation is a top-level flag, not a sub-command flag
$ vtctl -- --topo_implementation etcd2 AddCellInfo --root "/vitess/global"

New command line flags and behavior

vttablet --heartbeat_on_demand_duration

--heartbeat_on_demand_duration joins the already existing heartbeat flags --heartbeat_enable and --heartbeat_interval and adds new behavior to heartbeat writes.

--heartbeat_on_demand_duration takes a duration value, such as 5s.

The default value for --heartbeat_on_demand_duration is zero, which means the flag is not set and there is no change in behavior.

When --heartbeat_on_demand_duration has a positive value, then heartbeats are only injected on demand, based on internal requests. For example, when --heartbeat_on_demand_duration=5s, the tablet starts without injecting heartbeats.
An internal module, like the lag throttler, may request the heartbeat writer for heartbeats. Starting at that point in time, and for the duration (a lease) of 5s in our example, the tablet will write heartbeats.
If no other requests come in during that time, the tablet then ceases to write heartbeats. If more requests for heartbeats come in, the tablet extends the lease for the next 5s following each request.
It stops writing heartbeats 5s after the last request is received.

The heartbeats are generated according to --heartbeat_interval.

Deprecation of --online_ddl_check_interval

The flag --online_ddl_check_interval is deprecated and will be removed in v15. It has been unused in v13.

Removal of --gateway_implementation

In previous releases, the discoverygateway was deprecated. In Vitess 14 it is now entirely removed, along with the VTGate flag that allowed us to choose a gateway.

Deprecation of --planner_version

The flag --planner_version is deprecated and will be removed in v15.
Some binaries used --planner_version, and some used --planner-version.
This has been made consistent - all binaries that allow you to configure the planner now take --planner-version.
All uses of the underscore form have been deprecated and will be removed in v15.

Online DDL changes

Online DDL is generally available

Online DDL is no longer experimental (with the exception of pt-osc strategy). Specifically:

• Managed schema changes, the scheduler, the backing tables
• Supporting SQL syntax
• vitess strategy (online DDL via VReplication)
• gh-ost strategy (online DDL via 3rd party gh-ost)
• Recoverable migrations
• Revertible migrations
• Declarative migrations
• Postponed migrations
• And all other functionality

Are all considered production-ready.

pt-osc strategy (online DDL via 3rd party pt-online-schema-change) remains experimental.

ddl_strategy: 'vitess'

ddl_strategy now takes the value of vitess to indicate VReplication-based migrations. It is a synonym to online and uses the exact same functionality. The online term will be phased out in the future and vitess will remain the term of preference.

Example:

vtctlclient ApplySchema -skip_preflight -ddl_strategy='vitess' -sql "alter table my_table add column my_val int not null default 0" commerce

--singleton-context and REVERT migrations

It is now possible to submit a migration with --singleton-context strategy flag, while there's a pending (queued or running) REVERT migration that does not have a --singleton-context flag.

Support for CHECK constraints

Online DDL operations are more aware of CHECK constraints, and properly handle the limitation where a CHECK's name has to be unique in the schema. As opposed to letting MySQL choose arbitrary names for shadow table's CHECK constraints, Online DDL now generates unique yet deterministic names, such that all shards converge onto the same names.

Online DDL attempts to preserve the original check's name as a suffix to the generated name, where possible (names are limited to 64 characters).

Behavior changes

• vtctl ApplySchema --uuid_list='...' now rejects a migration if an existing migration has the same UUID but with different migration_context.

Table lifecycle

Views

Table lifecycle now supports views. It does not purge rows from views, and does not keep views in EVAC state (they are immediately transitioned to DROP state).

Fast drops

On Mysql 8.0.23 or later, the states PURGE and EVAC are automatically skipped, thanks to 8.0.23 improvements to DROP TABLE speed of operation.

Tablet throttler

API changes

Added /throttler/throttled-apps endpoint, which reports back all current throttling instructions. Note, this only reports explicit throttling requests (such as ones submitted by /throtler/throttle-app?app=...). It does not list incidental rejections based on throttle thresholds.

API endpoint /throttler/throttle-app now accepts a ratio query argument, a floating point value in the range [0..1], where:

• 0 means "do not throttle at all"
• 1 means "always throttle"
• Any number in between is allowed. For example, 0.3 means "throttle with 0.3 probability", i.e. for any given request there's a 30% chance that the request is denied. Overall we can expect ...

Vitess v14.0.0-rc1

Release of Vitess v14.0.0-RC1

Major Changes

Command-line syntax deprecations

Vitess has begun a transition to a new library for CLI flag parsing.
In order to facilitate a smooth transition, certain syntaxes that will not be supported in the future now issue deprecation warnings when used.

The messages you will likely see, along with explanations and migrations, are:

"Use of single-dash long flags is deprecated"

Single-dash usage will be only possible for short flags (e.g. -v is okay, but -verbose is not).

To migrate, update your CLI scripts from:

$ vttablet -tablet_alias zone1-100 -init_keyspace mykeyspace ... # old way

To:

$ vttablet --tablet_alias zone1-100 --init_keyspace mykeyspace ... # new way

"Detected a dashed argument after a position argument."

As the full deprecation text goes on to (attempt to) explain, mixing flags and positional arguments will change in a future version that will break scripts.

Currently, when invoking a binary like

$ vtctl --topo_implementation etcd2 AddCellInfo --root "/vitess/global"

everything after the AddCellInfo is treated by package flag as a positional argument, and we then use a sub FlagSet to parse flags specific to the subcommand.
So, at the top-level, flag.Args() returns ["AddCellInfo", "--root", "/vitess/global"].

The library we are transitioning to is more flexible, allowing flags and positional arguments to be interwoven on the command-line.
For the above example, this means that we would attempt to parse --root as a top-level flag for the vtctl binary.
This will cause the program to exit on error, because that flag is only defined on the AddCellInfo subcommand.

In order to transition, a standalone double-dash (literally, --) will cause the new flag library to treat everything following that as a positional argument, and also works with the current flag parsing code we use.

So, to transition the above example without breakage, update the command to:

$ vtctl --topo_implementation etcd2 AddCellInfo -- --root "/vitess/global"
$ # the following will also work
$ vtctl --topo_implementation etcd2 -- AddCellInfo --root "/vitess/global"
$ # the following will NOT work, because --topo_implementation is a top-level flag, not a sub-command flag
$ vtctl -- --topo_implementation etcd2 AddCellInfo --root "/vitess/global"

New command line flags and behavior

vttablet --heartbeat_on_demand_duration

--heartbeat_on_demand_duration joins the already existing heartbeat flags --heartbeat_enable and --heartbeat_interval and adds new behavior to heartbeat writes.

--heartbeat_on_demand_duration takes a duration value, such as 5s.

The default value for --heartbeat_on_demand_duration is zero, which means the flag is not set and there is no change in behavior.

When --heartbeat_on_demand_duration has a positive value, then heartbeats are only injected on demand, per internal requests. For example, when --heartbeat_on_demand_duration=5s, the tablet starts without injecting heartbeats. An internal module, like the lag throttle, may request the heartbeat writer for heartbeats. Starting at that point in time, and for the duration (a lease) of 5s in our example, the tablet will write heartbeats. If no other requests come in during that duration, then the tablet then ceases to write heartbeats. If more requests for heartbeats come while heartbeats are being written, then the tablet extends the lease for the next 5s following up each request. Thus, it stops writing heartbeats 5s after the last request is received.

The heartbeats are generated according to --heartbeat_interval.

Deprecation of --online_ddl_check_interval

The flag --online_ddl_check_interval is deprecated and will be removed in v15. It has been unused in v13.

Deprecation of --planner-version for vtexplain

The flag --planner-version is deprecated and will be removed in v15. Instead, please use --planer_version.

Online DDL changes

Online DDL is generally available

Online DDL is no longer experimental (with the exception of pt-osc strategy). Specifically:

• Managed schema changes, the scheduler, the backing tables
• Supporting SQL syntax
• vitess strategy (online DDL via VReplication)
• gh-ost strategy (online DDL via 3rd party gh-ost)
• Recoverable migrations
• Revertible migrations
• Declarative migrations
• Postponed migrations
• and all other functionality

Are all considered production-ready.

pt-osc strategy (online DDL via 3rd party pt-online-schema-change) remains experimental.

Throttling

See new SQL syntax for controlling/viewing throttling for Online DDL, down below.

ddl_strategy: 'vitess'

--ddl_strategy now takes the value of vitess to indicate VReplication-based migrations. It is a synonym to online and uses the exact same functionality. In the future, the online term will phase out, and vitess will remain the term of preference.

Example:

vtctlclient ApplySchema -- --skip_preflight --ddl_strategy='vitess' --sql "alter table my_table add column my_val int not null default 0" commerce

--singleton-context and REVERT migrations

It is now possible to submit a migration with --singleton-context strategy flag, while there's a pending (queued or running) REVERT migration that does not have a --singleton-context flag.

Support for CHECK constraints

Online DDL operations are more aware of CHECK constraints, and properly handle the limitation where a CHECK's name has to be unique in the schema. As opposed to letting MySQL choose arbitrary names for shadow table's CHECK consraints, Online DDL now generates unique yet deterministic names, such that all shards converge onto same names.

Online DDL attempts to preserve the original check's name as a suffix to the generated name, where possible (names are limited to 64 characters).

Behavior changes

• vtctlclient ApplySchema -- --uuid_list='...' now rejects a migration if an existing migration has the same UUID but with different migration_context.

Table lifecycle

Views

Table lifecycle now supports views. It ensures to not purge rows from views, and does not keep views in EVAC state (they are immediately transitioned to DROP state).

Fast drops

On Mysql 8.0.23 or later, the states PURGE and EVAC are automatically skipped, thanks to 8.0.23 improvement to DROP TABLE speed of operation.

Tablet throttler

API changes

Added /throttler/throttled-apps endpoint, which reports back all current throttling instructions. Note, this only reports explicit throttling requests (sych as ones submitted by /throtler/throttle-app?app=...). It does not list incidental rejections based on throttle thresholds.

API endpoint /throttler/throttle-app now accepts a ratio query argument, a floating point in the range [0..1], where:

• 0 means "do not throttle at all"
• 1 means "always throttle"
• any numbr in between is allowd. For example, 0.3 means "throttle in 0.3 probability", ie on a per request and based on a dice roll, there's a 30% change a request is denied. Overall we can expect about 30% of requests to be denied. Example: /throttler/throttle-app?app=vreplication&ratio=0.25

See new SQL syntax for controlling/viewing throttling, down below.

New Syntax

Control and view Online DDL throttling

We introduce the following syntax, to:

• Start/stop throttling for all Online DDL migrations, in general
• Start/stop throttling for a particular Online DDL migration
• View throttler state

ALTER VITESS_MIGRATION '<uuid>' THROTTLE [EXPIRE '<duration>'] [RATIO <ratio>];
ALTER VITESS_MIGRATION THROTTLE ALL [EXPIRE '<duration>'] [RATIO <ratio>];
ALTER VITESS_MIGRATION '<uuid>' UNTHROTTLE;
ALTER VITESS_MIGRATION UNTHROTTLE ALL;
SHOW VITESS_THROTTLED_APPS;

default duration is "infinite" (set as 100 years)

• allowed units are (s)ec, (m)in, (h)our
  ratio is in the range [0..1].
• 1 means full throttle - the app will not make any progress
• 0 means no throttling at all
• 0.8 means on 8 out of 10 checks the app makes, it gets refused

The syntax SHOW VITESS_THROTTLED_APPS is a generic call to the throttler, and returns information about all throttled apps, not specific to migrations

SHOW VITESS_MIGRATIONS ... output now includes user_throttle_ratio

This column is updated "once in a while", while a migration is running. Normally this is once a minute, but can be more frequent. The migration reports back what was the throttling instruction set by the user while it was/is running.
This column does not indicate any actual lag-based throttling that takes place per production state. It only reports the explicit throttling value set by the user.

Heartbeat

The throttler now checks in with the heartbeat writer to request heartbeats, any time it (the throttler) is asked for a check.

When --heartbeat_on_demand_duration is not set, there is no change in behavior.

When --heartbeat_on_demand_duration is set to a positive value, then the throttler ensures that the heartbeat writer generated heartbeats for at least the following duration. This also means at the first throttler check, it's possible that heartbeats are idle, and so the first check will fail. As heartbeats start running, followup checks will get a more accurate lag evaluation and will respond accordingly. In a sense, it's a "cold engine" scenario, where the engine takes time to start up, and then runs smoothly.

VDiff2

We introduced a new version of VDiff -- currently marked as Experimental -- that executes the VDiff on tablets rather than in vtctld. While this is experimental we encourage you to try it out ...

Vitess v12.0.4

Release of Vitess v12.0.4

Major Changes

Cluster management

A race condition #9819 that happens when running PlannedReparentShard was fixed through #9859.

Query Serving

Two major bugs on UNION are fixed via this release. The first one, #10257, solves an issue around the UNION executor,
where the execution context was not handled properly in a concurrent environment. The second one, #9945, was detected by
one of our UNION tests becoming flaky in the CI, it got solved by #9979, which focuses on improving the concurrency of UNION
executions.

A panic when ordering results in descending order on a hash vindex is also fixed. The original issue can be found here #10019.

Changelog

Bug fixes

Cluster management

• Fix the race between PromoteReplica and replication manager tick #9859

Query Serving

• Route explain table plan to the proper Keyspace #10028
• Multiple fixes to UNION planning and execution #10344

The release includes 5 commits (excluding merges)

Thanks to all our contributors: @GuptaManan100, @frouioui, @systay

Vitess v13.0.1

Release of Vitess v13.0.1

Changelog

Bug fixes

Cluster management

• Fix the race between PromoteReplica and replication manager tick #9859

Query Serving

• Fix __sq_has_values1 error with PulloutSubquery #9864
• Fix planner panic on derived tables sorting in query builder #9959
• Fix make concatenate and limit concurrent safe #9981
• Fix reserved connection retry logic when vttablet or mysql is down #10005
• Fix Gen4 sub query planning when the outer query is a dual query #10007
• Fix parsing of bind variables in a few places #10015
• Fix route explain tab plan to the proper Keyspace #10029
• Fix Sequence query to ignore reserved and transaction #10054
• Fix dual query with exists clause having system table query in it #10055
• Fix Gen4 only_full_group_by regression #10079

VReplication

• VPlayer use stored/binlogged ENUM index value in WHERE clauses #9868

CI/Build

Security

• Upgrade to go1.17.9 #10088
  The go1.17.9 version, released 2022-04-12, includes security fixes to the crypto/elliptic and encoding/pem packages, as well as bug fixes to the linker and runtime). More information here.

Enhancement

Query Serving

• Fix allow multiple distinct columns on single shards #10047
• Fix add parsing for NOW in DEFAULT clause #10085

The release includes 30 commits (excluding merges)

Thanks to all our contributors: @GuptaManan100, @frouioui, @harshit-gangal, @mattlord, @rohit-nayak-ps, @systay, @vmg

Vitess 13.0.0

Release of Vitess v13.0.0

Major Changes

Vitess now has native support for MySQL collations

When using the gen4 planner, Vitess is now capable of performing collation-aware string comparisons in the vtgates. This
improves the performance and reliability of several query plans that were previously relying on a debug-only SQL API in
MySQL to perform these comparisons remotely. It also enables new query plans that were previously not possible.

A full list of the supported collations can be
found in the Vitess documentation.

The native Evaluation engine in vtgate has been greatly improved

The SQL evaluation engine that runs inside the vtgates has been rewritten mostly from scratch to more closely match
MySQL's behavior. This allows Vitess to execute more parts of the query plans locally, and increases the complexity and
semantics of the SQL expressions that can be used to perform cross-shard queries.

vttablet -use_super_read_only flag now defaults to true

The default value used to be false. What this means is that during a failover, we will set super_read_only on database
flavors that support them (MySQL 5.7+ and Percona 5.7+). In addition, all Vitess-managed databases will be started
with super-read-only in the cnf file. It is expected that this change is safe and backwards-compatible. Anyone who is
relying on the current behavior should pass -use_super_read_only=false on the vttablet command line, and make sure
they are using a custom my.cnf instead of the one provided as the default by Vitess.

vtgate -buffer_implementation now defaults to keyspace_events

The default value used to be healthcheck. The new keyspace_events implementation has been tested in production with
good results and shows more consistent buffering behavior during PlannedReparentShard operations. The keyspace_events
implementation utilizes heuristics to detect additional cluster states where buffering is safe to perform, including
cases where the primary may be down. If there is a need to revert back to the previous buffer implementation, ensure
buffering is enabled in vtgate and pass the flag -buffer_implementation=healthcheck.

ddl_strategy: -postpone-completion flag

ddl_strategy (either @@ddl_strategy in VtGate or -ddl_strategy in vtctlclient ApplySchema) supports the
flag -postpone-completion

This flag indicates that the migration should not auto-complete. This applies for:

• any CREATE TABLE
• any DROP TABLE
• ALTER table in online strategy
• ALTER table in gh-ost strategy

Note that this flag is not supported for pt-osc strategy.

Behavior of migrations with this flag:

• an ALTER table begins, runs, but does not cut-over.
• CREATE or DROP migrations are silently not even scheduled

alter vitess_migration ... cleanup

A new query is supported:

alter vitess_migration '9748c3b7_7fdb_11eb_ac2c_f875a4d24e90' cleanup

This query tells Vitess that a migration's artifacts are good to be cleaned up asap. This allows Vitess to free disk
resources sooner. As a reminder, once a migration's artifacts are cleaned up, the migration is no longer revertible.

alter vitess_migration ... complete

A new query is supported:

alter vitess_migration '9748c3b7_7fdb_11eb_ac2c_f875a4d24e90' complete

This command indicates that a migration executed with -postpone-completion is good to complete. Behavior:

• For running ALTERs (online and gh-ost) which are ready to cut-over: cut-over imminently (though not immediately
  • cut-over depends on polling interval, replication lag, etc)
• For running ALTERs (online and gh-ost) which are only partly through the migration: they will cut-over
  automatically when they complete their work, as if -postpone-completion wasn't indicated
• For queued CREATE and DROP migrations: "unblock" them from being scheduled. They'll be scheduled at the scheduler'
  s discretion. there is no guarantee that they will be scheduled to run immediately.

vtctl/vtctlclient ApplySchema: ALTER VITESS_MIGRATION

vtctl ApplySchema now supports ALTER VITESS_MIGRATION ... statements. Example:

$ vtctl ApplySchema -skip_preflight -sql "alter vitess_migration '9748c3b7_7fdb_11eb_ac2c_f875a4d24e90' complete" commerce

vtctl/vtctlclient ApplySchema: allow zero in date

vtctl/vtctlclient ApplySchema now respects -allow-zero-in-date for direct strategy. For example, the following
statement is now accepted:

vtctlclient ApplySchema -skip_preflight -ddl_strategy='direct -allow-zero-in-date' -sql "create table if not exists t2(id int primary key, dt datetime default '0000-00-00 00:00:00')" commerce

vtctl/vtctlclient ApplySchema -uuid_list

vtctlient ApplySchema now support a new optional -uuid_list flag. It is possible for the user to explicitly specify
the UUIDs for given migration(s). UUIDs must be in a specific format. If given, number of UUIDs must match the number of
DDL statements. Example:

vtctlclient OnlineDDL ApplySchema -sql "drop table t1, drop table t2" -uuid_list "d08f0000_51c9_11ec_9cf2_0a43f95f28a3,d08f0001_51c9_11ec_9cf2_0a43f95f28a3" commerce

Vitess will assign each migration with given UUID in order of appearance. It is the user's responsibility to ensure
given UUIDs are globally unique. If the user submits a migration with an already existing UUID, that migration never
gets scheduled nor executed.

vtctl/vtctlclient ApplySchema -migration_context

-migration_context flag is synonymous to -request_context. Either will work. We will encourage use
of -migration_context as it is more consistent with output of SHOW VITESS_MIGRATIONS ... which includes
the migration_context column.

vtctl/vtctlclient ApplySchema -caller_id

-caller_id flag sets the Effective Caller ID of the ApplySchema operation so that the operation can succeed with a database that
is enforcing strict ACL checking.

vtctl/vtctlclient OnlineDDL ... complete

Complementing the alter vitess_migration ... complete query, a migration can also be completed via vtctl
or vtctlclient:

vtctlclient OnlineDDL <keyspace> complete <uuid>

For example:

vtctlclient OnlineDDL commerce complete d08ffe6b_51c9_11ec_9cf2_0a43f95f28a3

vtctl/vtctlclient OnlineDDL -json

The command now accepts an optional -json flag. With this flag, the output is a valid JSON listing all columns and
rows.

vtadmin-web updated to node v16.13.0 (LTS)

Building vtadmin-web now requires node >= v16.13.0 (LTS). Upgrade notes are given
in #9136.

PlannedReparentShard for cluster initialization

For setting up the cluster and electing a primary for the first time, PlannedReparentShard should be used
instead of InitShardPrimary.

InitShardPrimary is a forceful command and copies over the executed gtid set from the new primary to all the other replicas. So, if the user
isn't careful, it can lead to some replicas not being setup correctly and lead to errors in replication and recovery later.
PlannedReparentShard is a safer alternative and does not change the executed gtid set on the replicas forcefully. It is the preferred alternate to initialize
the cluster.

If using a custom init_db.sql that omits SET sql_log_bin = 0, then InitShardPrimary should still be used instead of PlannedReparentShard.

Durability Policy flag

A new flag has been added to vtctl, vtctld and vtworker binaries which allows the users to set the durability policies.

If semi-sync is not being used then -durability_policy should be set to none. This is also the default option.

If semi-sync is being used then -durability_policy should be set to semi_sync and -enable_semi_sync should be set in vttablets.

Incompatible Changes

Error message change when vttablet row limit exceeded:

• In previous Vitess versions, if the vttablet row limit (-queryserver-config-max-result-size) was exceeded, an error like:

  ERROR 10001 (HY000): target: unsharded.0.master: vttablet: rpc error: code = ResourceExhausted desc = Row count exceeded 10000 (errno 10001) (sqlstate HY000) ...
  would be reported to the client.

  To avoid confusion, the error mapping has been changed to report the error as similar to:

  ERROR 10001 (HY000): target: unsharded.0.primary: vttablet: rpc error: code = Aborted desc = Row count exceeded 10000 (errno 10001) (sqlstate HY000) ...
  instead

• Because of this error code change, the vttablet metric:
  vttablet_errors{error_code="ABORTED"}
  will be incremented upon this type of error, instead of the previous metric:
  vttablet_errors{error_code="RESOURCE_EXHAUSTED"}
• In addition, the vttablet error message logged is now different.
  Previously, a (misleading; due to the PoolFull) error was logged:

  E0112 09:48:25.420641  278125 tabletserver.go:1368] PoolFull: Row count exceeded 10000 (errno 10001) (sqlstate HY000) ...

  Post-change, a more accurate warning is logged instead:

  W0112 09:38:59.169264   35943 tabletserver.go:1503] Row count exceeded 10000 (errno 10001) (sqlstate HY000) ...

• If you were using -queryserver-config-terse-errors to redact some log messages containing bind vars in 13.0-SNAPSHOT, you should now instead enable -sanitize_log_messages which sanitizes all log messages containing sensitive info

Column types for textual queries now match MySQL's behavior

The column types for certain queries performed on a vtgate (most notably, those that SELECT system variables)
have been changed to match the types that would be returned if querying a MySQ...

Vitess 13.0.0-rc1

Release of Vitess v13.0.0-rc1

Major Changes

Vitess now has native support for MySQL collations

When using the gen4 planner, Vitess is now capable of performing collation-aware string comparisons in the vtgates. This
improves the performance and reliability of several query plans that were previously relying on a debug-only SQL API in
MySQL to perform these comparisons remotely. It also enables new query plans that were previously not possible.

A full list of the supported collations can be
found in the Vitess documentation.

The native Evaluation engine in vtgate has been greatly improved

The SQL evaluation engine that runs inside the vtgates has been rewritten mostly from scratch to more closely match
MySQL's behavior. This allows Vitess to execute more parts of the query plans locally, and increases the complexity and
semantics of the SQL expressions that can be used to perform cross-shard queries.

vttablet -use_super_read_only flag now defaults to true

The default value used to be false. What this means is that during a failover, we will set super_read_only on database
flavors that support them (MySQL 5.7+ and Percona 5.7+). In addition, all Vitess-managed databases will be started
with super-read-only in the cnf file. It is expected that this change is safe and backwards-compatible. Anyone who is
relying on the current behavior should pass -use_super_read_only=false on the vttablet command line, and make sure
they are using a custom my.cnf instead of the one provided as the default by Vitess.

vtgate -buffer_implementation now defaults to keyspace_events

The default value used to be healthcheck. The new keyspace_events implementation has been tested in production with
good results and shows more consistent buffering behavior during PlannedReparentShard operations. The keyspace_events
implementation utilizes heuristics to detect additional cluster states where buffering is safe to perform, including
cases where the primary may be down. If there is a need to revert back to the previous buffer implementation, ensure
buffering is enabled in vtgate and pass the flag -buffer_implementation=healthcheck.

ddl_strategy: -postpone-completion flag

ddl_strategy (either @@ddl_strategy in VtGate or -ddl_strategy in vtctlclient ApplySchema) supports the
flag -postpone-completion

This flag indicates that the migration should not auto-complete. This applies for:

• any CREATE TABLE
• any DROP TABLE
• ALTER table in online strategy
• ALTER table in gh-ost strategy

Note that this flag is not supported for pt-osc strategy.

Behavior of migrations with this flag:

• an ALTER table begins, runs, but does not cut-over.
• CREATE or DROP migrations are silently not even scheduled

alter vitess_migration ... cleanup

A new query is supported:

alter vitess_migration '9748c3b7_7fdb_11eb_ac2c_f875a4d24e90' cleanup

This query tells Vitess that a migration's artifacts are good to be cleaned up asap. This allows Vitess to free disk
resources sooner. As a reminder, once a migration's artifacts are cleaned up, the migration is no longer revertible.

alter vitess_migration ... complete

A new query is supported:

alter vitess_migration '9748c3b7_7fdb_11eb_ac2c_f875a4d24e90' complete

This command indicates that a migration executed with -postpone-completion is good to complete. Behavior:

• For running ALTERs (online and gh-ost) which are ready to cut-over: cut-over imminently (though not immediately
  • cut-over depends on polling interval, replication lag, etc)
• For running ALTERs (online and gh-ost) which are only partly through the migration: they will cut-over
  automatically when they complete their work, as if -postpone-completion wasn't indicated
• For queued CREATE and DROP migrations: "unblock" them from being scheduled. They'll be scheduled at the scheduler'
  s discretion. there is no guarantee that they will be scheduled to run immediately.

vtctl/vtctlclient ApplySchema: ALTER VITESS_MIGRATION

vtctl ApplySchema now supports ALTER VITESS_MIGRATION ... statements. Example:

$ vtctl ApplySchema -skip_preflight -sql "alter vitess_migration '9748c3b7_7fdb_11eb_ac2c_f875a4d24e90' complete" commerce

vtctl/vtctlclient ApplySchema: allow zero in date

vtctl/vtctlclient ApplySchema now respects -allow-zero-in-date for direct strategy. For example, the following
statement is now accepted:

vtctlclient ApplySchema -skip_preflight -ddl_strategy='direct -allow-zero-in-date' -sql "create table if not exists t2(id int primary key, dt datetime default '0000-00-00 00:00:00')" commerce

vtctl/vtctlclient ApplySchema -uuid_list

vtctlient ApplySchema now support a new optional -uuid_list flag. It is possible for the user to explicitly specify
the UUIDs for given migration(s). UUIDs must be in a specific format. If given, number of UUIDs must match the number of
DDL statements. Example:

vtctlclient OnlineDDL ApplySchema -sql "drop table t1, drop table t2" -uuid_list "d08f0000_51c9_11ec_9cf2_0a43f95f28a3,d08f0001_51c9_11ec_9cf2_0a43f95f28a3" commerce

Vitess will assign each migration with given UUID in order of appearance. It is the user's responsibility to ensure
given UUIDs are globally unique. If the user submits a migration with an already existing UUID, that migration never
gets scheduled nor executed.

vtctl/vtctlclient ApplySchema -migration_context

-migration_context flag is synonymous to -request_context. Either will work. We will encourage use
of -migration_context as it is more consistent with output of SHOW VITESS_MIGRATIONS ... which includes
the migration_context column.

vtctl/vtctlclient OnlineDDL ... complete

Complementing the alter vitess_migration ... complete query, a migration can also be completed via vtctl
or vtctlclient:

vtctlclient OnlineDDL <keyspace> complete <uuid>

For example:

vtctlclient OnlineDDL commerce complete d08ffe6b_51c9_11ec_9cf2_0a43f95f28a3

vtctl/vtctlclient OnlineDDL -json

The command now accepts an optional -json flag. With this flag, the output is a valid JSON listing all columns and
rows.

vtadmin-web updated to node v16.13.0 (LTS)

Building vtadmin-web now requires node >= v16.13.0 (LTS). Upgrade notes are given
in #9136.

PlannedReparentShard for cluster initialization

For setting up the cluster and electing a primary for the first time, PlannedReparentShard should be used
instead of InitShardPrimary.

InitShardPrimary is a forceful command and copies over the executed gtid set from the new primary to all the other replicas. So, if the user
isn't careful, it can lead to some replicas not being setup correctly and lead to errors in replication and recovery later.
PlannedReparentShard is a safer alternative and does not change the executed gtid set on the replicas forcefully. It is the preferred alternate to initialize
the cluster.

If using a custom init_db.sql that omits SET sql_log_bin = 0, then InitShardPrimary should still be used instead of PlannedReparentShard.

Durability Policy flag

A new flag has been added to vtctl, vtctld and vtworker binaries which allows the users to set the durability policies.

If semi-sync is not being used then -durability_policy should be set to none. This is also the default option.

If semi-sync is being used then -durability_policy should be set to semi_sync and -enable_semi_sync should be set in vttablets.

Incompatible Changes

Error message change when vttablet row limit exceeded:

• In previous Vitess versions, if the vttablet row limit (-queryserver-config-max-result-size) was exceeded, an error like:

  ERROR 10001 (HY000): target: unsharded.0.master: vttablet: rpc error: code = ResourceExhausted desc = Row count exceeded 10000 (errno 10001) (sqlstate HY000) ...
  would be reported to the client.

  To avoid confusion, the error mapping has been changed to report the error as similar to:

  ERROR 10001 (HY000): target: unsharded.0.primary: vttablet: rpc error: code = Aborted desc = Row count exceeded 10000 (errno 10001) (sqlstate HY000) ...
  instead

• Because of this error code change, the vttablet metric:
  vttablet_errors{error_code="ABORTED"}
  will be incremented upon this type of error, instead of the previous metric:
  vttablet_errors{error_code="RESOURCE_EXHAUSTED"}
• In addition, the vttablet error message logged is now different.
  Previously, a (misleading; due to the PoolFull) error was logged:

  E0112 09:48:25.420641  278125 tabletserver.go:1368] PoolFull: Row count exceeded 10000 (errno 10001) (sqlstate HY000) ...

  Post-change, a more accurate warning is logged instead:

  W0112 09:38:59.169264   35943 tabletserver.go:1503] Row count exceeded 10000 (errno 10001) (sqlstate HY000) ...

Column types for textual queries now match MySQL's behavior

The column types for certain queries performed on a vtgate (most notably, those that SELECT system variables)
have been changed to match the types that would be returned if querying a MySQL instance directly: textual fields that
were previously returned as VARBINARY will now appear as VARCHAR.

This change should not have an impact on MySQL clients/connectors for statically typed programming languages, as these
clients coerce the returned rows into whatever types the user has requested, but clients for dynamic programming
languages may now start returning as "string" values that were previously returned as "bytes".
...

Vitess 12.0.3

Release of Vitess v12.0.3

Announcement

This patch is providing an update regarding the Apache Log4j security vulnerability (CVE-2021-44832) (#9465).

---

Changelog

Dependabot

Java

• build(deps): bump log4j-api from 2.16.0 to 2.17.1 in /java #9465

The release includes 4 commits (excluding merges)

Thanks to all our contributors: @dbussink, @frouioui

Vitess 11.0.4

Release of Vitess v11.0.4

Announcement

This patch is providing an update regarding the Apache Log4j security vulnerability (CVE-2021-44832) (#9464).

Known Issues

• An issue where the value of the -force flag is used instead of -keep_data flag's value in v2 vreplication workflows (#9174) is known to be present in this release. A workaround is available in the description of issue #9174.

---

Changelog

Dependabot

Java

• build(deps): bump log4j-api from 2.16.0 to 2.17.1 in /java #9464

The release includes 6 commits (excluding merges)

Thanks to all our contributors: @dbussink, @frouioui