Releases: vitessio/vitess
Vitess v14.0.2
Release of Vitess v14.0.2
Known Issues
- Corrupted results for non-full-group-by queries with JOINs. This can be resolved by using full-group-by queries.
Major Changes
Upgrade to go1.18.5
Vitess v14.0.2
now runs on go1.18.5
.
The patch release of Go, go1.18.5
, was one of the main reasons for this release as it includes important security fixes to packages used by Vitess.
Below is a summary of this patch release. You can learn more here.
go1.18.4 (released 2022-07-12) includes security fixes to the compress/gzip, encoding/gob, encoding/xml, go/parser, io/fs, net/http, and path/filepath packages, as well as bug fixes to the compiler, the go command, the linker, the runtime, and the runtime/metrics package.
End of life of MariadDB 10.2
Since the end-of-life of MariaDB 10.2, its Docker image is unavailable, and we decided to remove the unit tests using this version of MariaDB. The Pull Request doing this change is available here.
You can find more information on the list of supported databases on our documentation website, here.
The entire changelog for this release can be found here.
The release includes 23 commits (excluding merges)
Thanks to all our contributors: @GuptaManan100, @dbussink, @frouioui, @mattlord, @shlomi-noach, @vitess-bot[bot]
Vitess v13.0.3
Release of Vitess v13.0.3
Major Changes
Upgrade to go1.17.13
Vitess v13.0.3
now runs on go1.17.13
.
The patch release of Go, go1.17.13
, was one of the main reasons for this release as it includes important security fixes to packages used by Vitess.
Below is a summary of this patch release. You can learn more here.
go1.17.13 (released 2022-08-01) includes security fixes to the encoding/gob and math/big packages, as well as bug fixes to the compiler and the runtime.
End of life of MariadDB 10.2
Since the end-of-life of MariaDB 10.2, its Docker image is unavailable, and we decided to remove the unit tests using this version of MariaDB. The Pull Request doing this change is available here.
This change is documented on our website here.
The entire changelog for this release can be found here.
The release includes 17 commits(excluding merges)
Thanks to all our contributors: @GuptaManan100, @dbussink, @frouioui, @harshit-gangal, @mattlord, @rohit-nayak-ps, @vitess-bot[bot]
Vitess v12.0.6
Release of Vitess v12.0.6
Major Changes
Upgrade to go1.17.13
Vitess v12.0.6
now runs on go1.17.13
.
The patch release of Go, go1.17.13
, was the main reason for this release as it includes important security fixes to packages used by Vitess.
Below is a summary of this patch release. You can learn more here.
go1.17.13 (released 2022-08-01) includes security fixes to the encoding/gob and math/big packages, as well as bug fixes to the compiler and the runtime.
End of life of MariadDB 10.2
Since the end-of-life of MariaDB 10.2, its Docker image is unavailable, and we decided to remove the unit tests using this version of MariaDB. The Pull Request doing this change is available here.
This change is documented on our website here.
The entire changelog for this release can be found here.
The release includes 11 commits (excluding merges)
Thanks to all our contributors: @GuptaManan100, @dbussink, @deepthi, @frouioui, @harshit-gangal, @rohit-nayak-ps
Vitess v12.0.5
Release of Vitess v12.0.5
Major Changes
Upgrade to go1.17.12
Vitess v12.0.5
now runs on go1.17.12
.
The patch release of Go, go1.17.12
, is the reason for this release as it includes important security fixes to packages used by Vitess.
Below is a summary of this patch release. You can learn more here.
go1.17.12 (released 2022-07-12) includes security fixes to the compress/gzip, encoding/gob, encoding/xml, go/parser, io/fs, net/http, and path/filepath packages, as well as bug fixes to the compiler, the go command, the runtime, and the runtime/metrics package. See the Go 1.17.12 milestone on our issue tracker for details.
The entire changelog for this release can be found here.
The release includes 7 commits (excluding merges)
Vitess v14.0.1
Release of Vitess v14.0.1
Known Issues
- Corrupted results for non-full-group-by queries with JOINs. This can be resolved by using full-group-by queries.
Major Changes
Upgrade to go1.18.4
Vitess v14.0.1
now runs on go1.18.4
.
The patch release of Go, go1.18.4
, was one of main motivations for this release as it includes important security fixes to packages used by Vitess.
Below is a summary of this patch release. You can learn more here.
go1.18.4 (released 2022-07-12) includes security fixes to the compress/gzip, encoding/gob, encoding/xml, go/parser, io/fs, net/http, and path/filepath packages, as well as bug fixes to the compiler, the go command, the linker, the runtime, and the runtime/metrics package. See the Go 1.18.4 milestone on our issue tracker for details.
The entire changelog for this release can be found here.
The release includes 25 commits (excluding merges)
Thanks to all our contributors: @GuptaManan100, @deepthi, @frouioui, @harshit-gangal, @mattlord, @rohit-nayak-ps, @shlomi-noach, @vitess-bot[bot]
Vitess v13.0.2
Release of Vitess v13.0.2
Major Changes
Upgrade to go1.17.12
Vitess v13.0.2
now runs on go1.17.12
.
The patch release of Go, go1.17.12
, was one of main motivations for this release as it includes important security fixes to packages used by Vitess.
Below is a summary of this patch release. You can learn more here.
go1.17.12 (released 2022-07-12) includes security fixes to the compress/gzip, encoding/gob, encoding/xml, go/parser, io/fs, net/http, and path/filepath packages, as well as bug fixes to the compiler, the go command, the runtime, and the runtime/metrics package. See the Go 1.17.12 milestone on our issue tracker for details.
The entire changelog for this release can be found here.
The release includes 28 commits (excluding merges)
Thanks to all our contributors: @GuptaManan100, @aquarapid, @frouioui, @harshit-gangal, @mattlord, @rohit-nayak-ps, @systay, @vitess-bot[bot], @vmg
Vitess v14.0.0
Release of Vitess v14.0.0
Summary
- Gen4 is now the default planner
- New query support
- Command-line syntax deprecations
- New command line flags and behavior
- Online DDL changes
- Table lifecycle
- Tablet throttler
- Heartbeat
- VDiff2
- Durability Policy
- Deprecation of Durability Configuration
- Advisory locking optimizations
- Pre-Legacy Resharding is now deprecated
Known Issues
- VTOrc doesn't discover the tablets of a keyspace if the durability policy doesn't exist in the topo server when it comes up. This can be resolved by restarting VTOrc.
- Corrupted results for non-full-group-by queries with JOINs. This can be resolved by using full-group-by queries.
Major Changes
Gen4 is now the default planner
The new planner has been in the works since end of 2020, and it's finally grown enough to be able to become the default planner for Vitess.
This means that many more queries are supported on sharded keyspaces, and old queries might get planned better than before.
You can always roll back to the earlier planner, either by providing the flag --planner-version=V3
to vtgate
, or by adding a comment to individual queries, like so:
select /*vt+ PLANNER=V3 */ name, count(*) from users
New query support
Support for aggregation across shards
Vitess can now plan and execute most aggregation queries across multiple shards and/or keyspaces.
INSERT from SELECT
Support has been added for inserting new data from SELECT queries.
Now you can insert data from a query into a table using a query like:
insert into tbl (col) select id from users
UPDATE from SELECT
Similarly, we have added support for UPDATE with scalar sub-queries. This allows for queries where the updated value is fetched using a subquery, such as this example:
update tbl set foo = (select count(*) from otherTbl)
Command-line syntax deprecations
Vitess has begun a transition to a new library for CLI flag parsing.
In order to facilitate a smooth transition, certain syntaxes that will not be supported in the future now issue deprecation warnings when used.
The messages you will likely see, along with explanations and migrations, are:
"Use of single-dash long flags is deprecated"
Single-dash usage will be only possible for short flags (e.g. -v
is okay, but -verbose
is not).
To migrate, update your CLI scripts from:
$ vttablet -tablet_alias zone1-100 -init_keyspace mykeyspace ... # old way
To:
$ vttablet --tablet_alias zone1-100 --init_keyspace mykeyspace ... # new way
"Detected a dashed argument after a position argument."
As the full deprecation text goes on to (attempt to) explain, mixing flags and positional arguments will change in a future version that will break scripts.
Currently, when invoking a binary like:
$ vtctl --topo_implementation etcd2 AddCellInfo --root "/vitess/global"
Everything after the AddCellInfo
is treated by package flag
as a positional argument, and we then use a sub FlagSet to parse flags specific to the subcommand.
So, at the top-level, flag.Args()
returns ["AddCellInfo", "--root", "/vitess/global"]
.
The library we are transitioning to is more flexible, allowing flags and positional arguments to be interwoven on the command-line.
For the above example, this means that we would attempt to parse --root
as a top-level flag for the VTCtl
binary.
This will cause the program to exit on error, because that flag is only defined on the AddCellInfo
subcommand.
In order to transition, a standalone double-dash (literally, --
) will cause the new flag library to treat everything following that as a positional argument, and also works with the current flag parsing code we use.
So, to transition the above example without breakage, update the command to:
$ vtctl --topo_implementation etcd2 AddCellInfo -- --root "/vitess/global"
$ # the following will also work
$ vtctl --topo_implementation etcd2 -- AddCellInfo --root "/vitess/global"
$ # the following will NOT work, because --topo_implementation is a top-level flag, not a sub-command flag
$ vtctl -- --topo_implementation etcd2 AddCellInfo --root "/vitess/global"
New command line flags and behavior
vttablet --heartbeat_on_demand_duration
--heartbeat_on_demand_duration
joins the already existing heartbeat flags --heartbeat_enable
and --heartbeat_interval
and adds new behavior to heartbeat writes.
--heartbeat_on_demand_duration
takes a duration value, such as 5s
.
The default value for --heartbeat_on_demand_duration
is zero, which means the flag is not set and there is no change in behavior.
When --heartbeat_on_demand_duration
has a positive value, then heartbeats are only injected on demand, based on internal requests. For example, when --heartbeat_on_demand_duration=5s
, the tablet starts without injecting heartbeats.
An internal module, like the lag throttler, may request the heartbeat writer for heartbeats. Starting at that point in time, and for the duration (a lease) of 5s
in our example, the tablet will write heartbeats.
If no other requests come in during that time, the tablet then ceases to write heartbeats. If more requests for heartbeats come in, the tablet extends the lease for the next 5s
following each request.
It stops writing heartbeats 5s
after the last request is received.
The heartbeats are generated according to --heartbeat_interval
.
Deprecation of --online_ddl_check_interval
The flag --online_ddl_check_interval
is deprecated and will be removed in v15
. It has been unused in v13
.
Removal of --gateway_implementation
In previous releases, the discoverygateway
was deprecated. In Vitess 14 it is now entirely removed, along with the VTGate flag that allowed us to choose a gateway.
Deprecation of --planner_version
The flag --planner_version
is deprecated and will be removed in v15
.
Some binaries used --planner_version
, and some used --planner-version
.
This has been made consistent - all binaries that allow you to configure the planner now take --planner-version
.
All uses of the underscore form have been deprecated and will be removed in v15
.
Online DDL changes
Online DDL is generally available
Online DDL is no longer experimental (with the exception of pt-osc
strategy). Specifically:
- Managed schema changes, the scheduler, the backing tables
- Supporting SQL syntax
vitess
strategy (online DDL via VReplication)gh-ost
strategy (online DDL via 3rd partygh-ost
)- Recoverable migrations
- Revertible migrations
- Declarative migrations
- Postponed migrations
- And all other functionality
Are all considered production-ready.
pt-osc
strategy (online DDL via 3rd party pt-online-schema-change
) remains experimental.
ddl_strategy: 'vitess'
ddl_strategy
now takes the value of vitess
to indicate VReplication-based migrations. It is a synonym to online
and uses the exact same functionality. The online
term will be phased out in the future and vitess
will remain the term of preference.
Example:
vtctlclient ApplySchema -skip_preflight -ddl_strategy='vitess' -sql "alter table my_table add column my_val int not null default 0" commerce
--singleton-context and REVERT migrations
It is now possible to submit a migration with --singleton-context
strategy flag, while there's a pending (queued or running) REVERT
migration that does not have a --singleton-context
flag.
Support for CHECK constraints
Online DDL operations are more aware of CHECK
constraints, and properly handle the limitation where a CHECK
's name has to be unique in the schema. As opposed to letting MySQL choose arbitrary names for shadow table's CHECK
constraints, Online DDL now generates unique yet deterministic names, such that all shards converge onto the same names.
Online DDL attempts to preserve the original check's name as a suffix to the generated name, where possible (names are limited to 64
characters).
Behavior changes
vtctl ApplySchema --uuid_list='...'
now rejects a migration if an existing migration has the same UUID but with differentmigration_context
.
Table lifecycle
Views
Table lifecycle now supports views. It does not purge rows from views, and does not keep views in EVAC
state (they are immediately transitioned to DROP
state).
Fast drops
On Mysql 8.0.23
or later, the states PURGE
and EVAC
are automatically skipped, thanks to 8.0.23
improvements to DROP TABLE
speed of operation.
Tablet throttler
API changes
Added /throttler/throttled-apps
endpoint, which reports back all current throttling instructions. Note, this only reports explicit throttling requests (such as ones submitted by /throtler/throttle-app?app=...
). It does not list incidental rejections based on throttle thresholds.
API endpoint /throttler/throttle-app
now accepts a ratio
query argument, a floating point value in the range [0..1]
, where:
0
means "do not throttle at all"1
means "always throttle"- Any number in between is allowed. For example,
0.3
means "throttle with 0.3 probability", i.e. for any given request there's a 30% chance that the request is denied. Overall we can expect ...
Vitess v14.0.0-rc1
Release of Vitess v14.0.0-RC1
Major Changes
Command-line syntax deprecations
Vitess has begun a transition to a new library for CLI flag parsing.
In order to facilitate a smooth transition, certain syntaxes that will not be supported in the future now issue deprecation warnings when used.
The messages you will likely see, along with explanations and migrations, are:
"Use of single-dash long flags is deprecated"
Single-dash usage will be only possible for short flags (e.g. -v
is okay, but -verbose
is not).
To migrate, update your CLI scripts from:
$ vttablet -tablet_alias zone1-100 -init_keyspace mykeyspace ... # old way
To:
$ vttablet --tablet_alias zone1-100 --init_keyspace mykeyspace ... # new way
"Detected a dashed argument after a position argument."
As the full deprecation text goes on to (attempt to) explain, mixing flags and positional arguments will change in a future version that will break scripts.
Currently, when invoking a binary like
$ vtctl --topo_implementation etcd2 AddCellInfo --root "/vitess/global"
everything after the AddCellInfo
is treated by package flag
as a positional argument, and we then use a sub FlagSet to parse flags specific to the subcommand.
So, at the top-level, flag.Args()
returns ["AddCellInfo", "--root", "/vitess/global"]
.
The library we are transitioning to is more flexible, allowing flags and positional arguments to be interwoven on the command-line.
For the above example, this means that we would attempt to parse --root
as a top-level flag for the vtctl
binary.
This will cause the program to exit on error, because that flag is only defined on the AddCellInfo
subcommand.
In order to transition, a standalone double-dash (literally, --
) will cause the new flag library to treat everything following that as a positional argument, and also works with the current flag parsing code we use.
So, to transition the above example without breakage, update the command to:
$ vtctl --topo_implementation etcd2 AddCellInfo -- --root "/vitess/global"
$ # the following will also work
$ vtctl --topo_implementation etcd2 -- AddCellInfo --root "/vitess/global"
$ # the following will NOT work, because --topo_implementation is a top-level flag, not a sub-command flag
$ vtctl -- --topo_implementation etcd2 AddCellInfo --root "/vitess/global"
New command line flags and behavior
vttablet --heartbeat_on_demand_duration
--heartbeat_on_demand_duration
joins the already existing heartbeat flags --heartbeat_enable
and --heartbeat_interval
and adds new behavior to heartbeat writes.
--heartbeat_on_demand_duration
takes a duration value, such as 5s
.
The default value for --heartbeat_on_demand_duration
is zero, which means the flag is not set and there is no change in behavior.
When --heartbeat_on_demand_duration
has a positive value, then heartbeats are only injected on demand, per internal requests. For example, when --heartbeat_on_demand_duration=5s
, the tablet starts without injecting heartbeats. An internal module, like the lag throttle, may request the heartbeat writer for heartbeats. Starting at that point in time, and for the duration (a lease) of 5s
in our example, the tablet will write heartbeats. If no other requests come in during that duration, then the tablet then ceases to write heartbeats. If more requests for heartbeats come while heartbeats are being written, then the tablet extends the lease for the next 5s
following up each request. Thus, it stops writing heartbeats 5s
after the last request is received.
The heartbeats are generated according to --heartbeat_interval
.
Deprecation of --online_ddl_check_interval
The flag --online_ddl_check_interval
is deprecated and will be removed in v15
. It has been unused in v13
.
Deprecation of --planner-version for vtexplain
The flag --planner-version
is deprecated and will be removed in v15
. Instead, please use --planer_version
.
Online DDL changes
Online DDL is generally available
Online DDL is no longer experimental (with the exception of pt-osc
strategy). Specifically:
- Managed schema changes, the scheduler, the backing tables
- Supporting SQL syntax
vitess
strategy (online DDL via VReplication)gh-ost
strategy (online DDL via 3rd partygh-ost
)- Recoverable migrations
- Revertible migrations
- Declarative migrations
- Postponed migrations
- and all other functionality
Are all considered production-ready.
pt-osc
strategy (online DDL via 3rd party pt-online-schema-change
) remains experimental.
Throttling
See new SQL syntax for controlling/viewing throttling for Online DDL, down below.
ddl_strategy: 'vitess'
--ddl_strategy
now takes the value of vitess
to indicate VReplication-based migrations. It is a synonym to online
and uses the exact same functionality. In the future, the online
term will phase out, and vitess
will remain the term of preference.
Example:
vtctlclient ApplySchema -- --skip_preflight --ddl_strategy='vitess' --sql "alter table my_table add column my_val int not null default 0" commerce
--singleton-context and REVERT migrations
It is now possible to submit a migration with --singleton-context
strategy flag, while there's a pending (queued or running) REVERT
migration that does not have a --singleton-context
flag.
Support for CHECK constraints
Online DDL operations are more aware of CHECK
constraints, and properly handle the limitation where a CHECK
's name has to be unique in the schema. As opposed to letting MySQL choose arbitrary names for shadow table's CHECK
consraints, Online DDL now generates unique yet deterministic names, such that all shards converge onto same names.
Online DDL attempts to preserve the original check's name as a suffix to the generated name, where possible (names are limited to 64
characters).
Behavior changes
vtctlclient ApplySchema -- --uuid_list='...'
now rejects a migration if an existing migration has the same UUID but with differentmigration_context
.
Table lifecycle
Views
Table lifecycle now supports views. It ensures to not purge rows from views, and does not keep views in EVAC
state (they are immediately transitioned to DROP
state).
Fast drops
On Mysql 8.0.23
or later, the states PURGE
and EVAC
are automatically skipped, thanks to 8.0.23
improvement to DROP TABLE
speed of operation.
Tablet throttler
API changes
Added /throttler/throttled-apps
endpoint, which reports back all current throttling instructions. Note, this only reports explicit throttling requests (sych as ones submitted by /throtler/throttle-app?app=...
). It does not list incidental rejections based on throttle thresholds.
API endpoint /throttler/throttle-app
now accepts a ratio
query argument, a floating point in the range [0..1]
, where:
0
means "do not throttle at all"1
means "always throttle"- any numbr in between is allowd. For example,
0.3
means "throttle in 0.3 probability", ie on a per request and based on a dice roll, there's a30%
change a request is denied. Overall we can expect about30%
of requests to be denied. Example:/throttler/throttle-app?app=vreplication&ratio=0.25
See new SQL syntax for controlling/viewing throttling, down below.
New Syntax
Control and view Online DDL throttling
We introduce the following syntax, to:
- Start/stop throttling for all Online DDL migrations, in general
- Start/stop throttling for a particular Online DDL migration
- View throttler state
ALTER VITESS_MIGRATION '<uuid>' THROTTLE [EXPIRE '<duration>'] [RATIO <ratio>];
ALTER VITESS_MIGRATION THROTTLE ALL [EXPIRE '<duration>'] [RATIO <ratio>];
ALTER VITESS_MIGRATION '<uuid>' UNTHROTTLE;
ALTER VITESS_MIGRATION UNTHROTTLE ALL;
SHOW VITESS_THROTTLED_APPS;
default duration
is "infinite" (set as 100 years)
- allowed units are (s)ec, (m)in, (h)our
ratio is in the range[0..1]
. 1
means full throttle - the app will not make any progress0
means no throttling at all0.8
means on 8 out of 10 checks the app makes, it gets refused
The syntax SHOW VITESS_THROTTLED_APPS
is a generic call to the throttler, and returns information about all throttled apps, not specific to migrations
SHOW VITESS_MIGRATIONS ...
output now includes user_throttle_ratio
This column is updated "once in a while", while a migration is running. Normally this is once a minute, but can be more frequent. The migration reports back what was the throttling instruction set by the user while it was/is running.
This column does not indicate any actual lag-based throttling that takes place per production state. It only reports the explicit throttling value set by the user.
Heartbeat
The throttler now checks in with the heartbeat writer to request heartbeats, any time it (the throttler) is asked for a check.
When --heartbeat_on_demand_duration
is not set, there is no change in behavior.
When --heartbeat_on_demand_duration
is set to a positive value, then the throttler ensures that the heartbeat writer generated heartbeats for at least the following duration. This also means at the first throttler check, it's possible that heartbeats are idle, and so the first check will fail. As heartbeats start running, followup checks will get a more accurate lag evaluation and will respond accordingly. In a sense, it's a "cold engine" scenario, where the engine takes time to start up, and then runs smoothly.
VDiff2
We introduced a new version of VDiff -- currently marked as Experimental -- that executes the VDiff on tablets rather than in vtctld. While this is experimental we encourage you to try it out ...
Vitess v12.0.4
Release of Vitess v12.0.4
Major Changes
Cluster management
A race condition #9819 that happens when running PlannedReparentShard
was fixed through #9859.
Query Serving
Two major bugs on UNION
are fixed via this release. The first one, #10257, solves an issue around the UNION
executor,
where the execution context was not handled properly in a concurrent environment. The second one, #9945, was detected by
one of our UNION
tests becoming flaky in the CI, it got solved by #9979, which focuses on improving the concurrency of UNION
executions.
A panic when ordering results in descending order on a hash
vindex is also fixed. The original issue can be found here #10019.
Changelog
Bug fixes
Cluster management
- Fix the race between PromoteReplica and replication manager tick #9859
Query Serving
- Route explain table plan to the proper Keyspace #10028
- Multiple fixes to UNION planning and execution #10344
The release includes 5 commits (excluding merges)
Thanks to all our contributors: @GuptaManan100, @frouioui, @systay
Vitess v13.0.1
Release of Vitess v13.0.1
Changelog
Bug fixes
Cluster management
- Fix the race between PromoteReplica and replication manager tick #9859
Query Serving
- Fix
__sq_has_values1
error withPulloutSubquery
#9864 - Fix planner panic on derived tables sorting in query builder #9959
- Fix make concatenate and limit concurrent safe #9981
- Fix reserved connection retry logic when vttablet or mysql is down #10005
- Fix Gen4 sub query planning when the outer query is a dual query #10007
- Fix parsing of bind variables in a few places #10015
- Fix route explain tab plan to the proper Keyspace #10029
- Fix Sequence query to ignore reserved and transaction #10054
- Fix dual query with exists clause having system table query in it #10055
- Fix Gen4 only_full_group_by regression #10079
VReplication
- VPlayer use stored/binlogged ENUM index value in WHERE clauses #9868
CI/Build
Security
- Upgrade to
go1.17.9
#10088
Thego1.17.9
version, released 2022-04-12, includes security fixes to thecrypto/elliptic
andencoding/pem
packages, as well as bug fixes to the linker and runtime). More information here.
Enhancement
Query Serving
- Fix allow multiple distinct columns on single shards #10047
- Fix add parsing for NOW in DEFAULT clause #10085
The release includes 30 commits (excluding merges)
Thanks to all our contributors: @GuptaManan100, @frouioui, @harshit-gangal, @mattlord, @rohit-nayak-ps, @systay, @vmg