Vitess v15.0.3

Release of Vitess v15.0.3

The entire changelog for this release can be found here.

The release includes 52 commits (excluding merges)

Thanks to all our contributors: @GuptaManan100, @ajm188, @dbussink, @deepthi, @frouioui, @harshit-gangal, @mattlord, @rsajwani, @shlomi-noach, @systay, @vitess-bot[bot], @vmg

Vitess v14.0.5

Release of Vitess v14.0.5

The entire changelog for this release can be found here.

The release includes 33 commits (excluding merges)

Thanks to all our contributors: @GuptaManan100, @dbussink, @frouioui, @harshit-gangal, @rsajwani, @shlomi-noach, @systay, @vitess-bot[bot], @vmg

Vitess v16.0.0

Release of Vitess v16.0.0

Summary

Table of Contents

• Known Issues
  • MySQL & Xtrabackup known issue
  • VTTablet Restore Metrics
  • Schema-initialization stuck on semi-sync ACKs while upgrading to v16.0.0
  • Broken downgrade from v17.x.x when super_read_only turned on by default
• Major Changes
  • Breaking Changes
    • VTGate Advertised MySQL Version
    • Default MySQL version on Docker
    • ⚠️ Upgrading to this release with vitess-operator
    • Flag Deletions and Deprecations
      • VTCtld
      • MySQLCtl
      • VTTablet
      • VTBackup
      • VTOrc
    • lock-timeout and remote_operation_timeout Changes
    • Orchestrator Integration Deletion
    • vtctld UI Removal
    • Query Serving Errors
    • Logstats Table and Keyspace removed
    • Removed Stats
    • Deprecated Stats
    • Normalized labels in the Prometheus Exporter
  • Replication manager removal and VTOrc becomes mandatory
  • VReplication
    • VStream Copy Resume
    • VDiff2 GA
  • Tablet throttler
  • Incremental backup and point in time recovery
  • New command line flags and behavior
    • VTGate: Support query timeout --query-timeout
    • VTTablet: VReplication parallel insert workers --vreplication-parallel-insert-workers
    • VTTablet: --queryserver-config-pool-conn-max-lifetime
    • vttablet --throttler-config-via-topo
    • vtctldclient UpdateThrottlerConfig
    • vtctldclient Backup --incremental_from_pos
    • vtctldclient RestoreFromBackup --restore_to_pos
    • New vexplain command
  • Important bug fixes
    • Corrupted results for non-full-group-by queries with JOINs
  • Deprecations and Removals
  • MySQL Compatibility
    • Transaction Isolation Level
    • Transaction Access Mode
    • Support for views
  • VTTestServer
    • Performance Improvement
• Minor Changes
  • Backup compression benchmarks
• Refactor
  • VTTablet sidecar schema maintenance refactor

Known Issues

MySQL & Xtrabackup known issue

There is a known issue with MySQL's INSTANT DDL combined with Percona XtraBackup, that affects users of Vitess 16.0.
The problem is described in https://docs.percona.com/percona-xtrabackup/8.0/em/instant.html, and the immediate impact is you may not be able to backup your database using XtraBackup under certain conditions.

As of MySQL 8.0.12, the default ALGORITHM for InnoDB's ALTER TABLE is INSTANT. In 8.0.12 only a small number of operations were eligible for INSTANT, but MySQL 8.0.29 added support for more common cases.
Unfortunately, the changes in 8.0.29 affect XtraBackup as follows: if you ALTER TABLE in MySQL 8.0.29, and that ALTER is eligible for INSTANT DDL (e.g. add new table column), then as of that moment, XtraBackup is unable to backup that table, hence your entire database.

It is important to note that even if you then upgrade your MySQL server to, e.g. 8.0.32, the table still cannot be backed up.

Versions where XtraBackup is unable to backup such tables: MySQL 8.0.29 - 8.0.31. This does not apply to Percona Server flavor.

The issue is resolved with Percona XtraBackup 8.0.32 combined with MySQL 8.0.32.

You might be affected if:

• You're using MySQL 8.0.29 - 8.0.31 and are using XtraBackup to backup your database
• and, you have issued an ALTER TABLE, either directly, or using Online DDL in vitess v16.0 and below

A futures Vitess patch release v16.0.1 will address the issue via Online DDL migrations.

Mitigations

• Use Percona XtraBackup 8.0.32 combined with MySQL 8.0.32. To go with this option, you can use the docker image vitess/lite:v16.0.0-mysql-8.0.32.
• or, Use a Percona Server flavor
• or, always ensure to add ALGORITHM=INPLACE or ALGORITHM=COPY to your ALTER TABLE statements

Workarounds

If you have already been affected, these are the options to be able to backup your database:

• Use builtin backups, see https://vitess.io/docs/15.0/user-guides/operating-vitess/backup-and-restore/creating-a-backup/. builting backups are not based on XtraBackup.
• Upgrade to MySQL 8.0.32 or above and to Xtrabackup 8.0.32, or switch to Percona Server. To go with this option, you can use the docker image vitess/lite:v16.0.0-mysql-8.0.32. Then rebuild the table directly via:
  • OPTIMIZE TABLE your_table
  • or, ALTER TABLE your_table ENGINE=INNOB
• Upgrade to Vitess patch release v16.0.1, upgrade to MySQL 8.0.32 or above and to Xtrabackup 8.0.32, or switch to Percona Server, and rebuild the table via Online DDL:

$ vtctldclient ApplySchema --skip_preflight --ddl_strategy "vitess" --sql "ALTER TABLE your_table ENGINE=InnoDB" your_keyspace

or

> SET @@ddl_strategy='vitess';
> ALTER TABLE your_table ENGINE=InnoDB;

VTTablet Restore Metrics

As part of the VTTablet Sidecar Schema Maintenance Refactor in v16.0.0, we dropped the local_metadata table from the sidecar database schema. This table was storing a couple of metrics related to restores from backup, which have now been lost.
They have been re-introduced in v17.0.0 as metrics that can be accessed from /debug/vars.

The original issue can be found here.

Schema-initialization stuck on semi-sync ACKs while upgrading to v16.0.0

During upgrades from <= v15.x.x to v16.0.0, as part of PromoteReplica call, the schema-init realizes that there are schema diffs to apply and ends up writing to the database.
The issue is that if semi-sync is enabled, all of these writes get blocked indefinitely.
Eventually, PromoteReplica fails, and this fails the entire PRS call.

A fix for this issue was merged on release-16.0 in PR#13441, read the corresponding bug report to learn more.

This issue is fixed in v16.0.3 and later patch releases.

Broken downgrade from v17.x.x when super_read_only turned on by default

In v17.x.x super_read_only is turned on by default meaning that downgrading from v17 to v16.0.0 breaks due to init_db.sql needing write access.

This issue is fixed in >= v16.0.3 thanks to PR #13525

Major Changes

Breaking Changes

VTGate Advertised MySQL Version

Since Pull Request #11989, VTGate advertises MySQL version 8.0.30. This is a breaking change for clients that rely on the VTGate advertised MySQL version and still use MySQL 5.7.
The users can set the mysql_server_version flag to advertise the correct version.

It is worth noting that the feature to avoid using reserved connections depends on the mysql_server_version CLI flag, which default value has been changed from 5.7.9-vitess to 8.0.30-vitess. We recommend that users running MySQL 5.7 set vtgate's mysql_server_version CLI flag to 5.7.9-vitess to prevent the queries from being unexpectedly rewritten.

Default MySQL version on Docker

The default major MySQL version used by our vitess/lite:latest image is going from 5.7 to 8.0. Additionally, the patch version of MySQL80 has been upgraded from 8.0.23 to 8.0.30.
This change was brought by Pull Request #12252.

⚠️Upgrading to this release with vitess-operator

If you are using the vitess-operator and want to remain on MySQL 5.7, you are required to use the vitess/lite:v16.0.0-mysql57 Docker Image, otherwise the vitess/lite:v16.0.0 image will be on MySQL 80.

However, if you are running MySQL 8.0 on the vitess-operator, with for instance vitess/lite:v15.0.2-mysql80, considering that we are bumping the patch version of MySQL 80 from 8.0.23 to 8.0.30, you will have to manually upgrade:

1. Add innodb_fast_shutdown=0 to your extra cnf in your YAML file.
2. Apply this file.
3. Wait for all the pods to be health...

Vitess v16.0.0-rc1

Release of Vitess v16.0.0-rc1

Summary

Table of Contents

• Major Changes
  • VReplication
    • VStream Copy Resume
    • VDiff2 GA
  • Tablet throttler
  • Incremental backup and point in time recovery
  • Replication manager removal and VTOrc becomes mandatory
  • Breaking Changes
    • VTGate Advertised MySQL Version
    • Default MySQL version on Docker
    • Running Vitess on the Operator
    • vtctld UI Removal
    • vtctld Flag Deprecation & Deletions
    • Orchestrator Integration Deletion
    • mysqlctl Flags
    • Query Serving Errors
    • Logstats Table and Keyspace removed
    • Removed Stats
    • Deprecated Stats
    • Removed flag
    • lock-timeout and remote_operation_timeout Changes
    • Normalized labels in the Prometheus Exporter
  • New command line flags and behavior
    • VTGate: Support query timeout --query-timeout
    • VTTablet: VReplication parallel insert workers --vreplication-parallel-insert-workers
    • VTTablet: --queryserver-config-pool-conn-max-lifetime
    • vttablet --throttler-config-via-topo
    • vtctldclient UpdateThrottlerConfig
    • vtctldclient Backup --incremental_from_pos
    • vtctldclient RestoreFromBackup --restore_to_pos
    • New vexplain command
  • Important bug fixes
    • Corrupted results for non-full-group-by queries with JOINs
  • Deprecations and Removals
  • MySQL Compatibility
    • Transaction Isolation Level
    • Transaction Access Mode
    • Support for views
  • VTOrc
    • Flag Deprecations
  • VTTestServer
    • Performance Improvement
• Minor Changes
  • Backup compression benchmarks
• Refactor
  • VTTablet sidecar schema maintenance refactor

Major Changes

VReplication

VStream Copy Resume

In PR #11103 we introduced the ability to resume a VTGate VStream copy operation. This is useful when a VStream copy operation is interrupted due to e.g. a network failure or a server restart. The VStream copy operation can be resumed by specifying each table's last seen primary key value in the VStream request. Please see the VStream docs for more details.

VDiff2 GA

We are marking VDiff v2 as Generally Available or production-ready in v16. We now recommend that you use v2 rather than v1 going forward. V1 will be deprecated and eventually removed in future releases.
If you wish to use v1 for any reason, you will now need to specify the --v1 flag.

Tablet throttler

The tablet throttler can now be configured dynamically. Configuration is now found in the topo service, and applies to all tablets in all shards and cells of a given keyspace. For backwards compatibility v16 still supports vttablet-based command line flags for throttler ocnfiguration.

It is possible to enable/disable, to change throttling threshold as well as the throttler query.

See #11604

Incremental backup and point in time recovery

In PR #11097 we introduced native incremental backup and point in time recovery:

• It is possible to take an incremental backup, starting with last known (full or incremental) backup, and up to either a specified (GTID) position, or current ("auto") position.
• The backup is done by copying binary logs. The binary logs are rotated as needed.
• It is then possible to restore a backup up to a given point in time (GTID position). This involves finding a restore path consisting of a full backup and zero or more incremental backups, applied up to the given point in time.
• A server restored to a point in time remains in DRAINED tablet type, and does not join the replication stream (thus, "frozen" in time).
• It is possible to take incremental backups from different tablets. It is OK to have overlaps in incremental backup contents. The restore process chooses a valid path, and is valid as long as there are no gaps in the backed up binary log content.

Replication manager removal and VTOrc becomes mandatory

VTOrc is now a required component of Vitess starting from v16. If the users want VTOrc to manage replication, then they must run VTOrc.
Replication manager is removed from vttablets since the responsibility of fixing replication lies entirely with VTOrc now.
The flag disable-replication-manager is deprecated and will be removed in a later release.

Breaking Changes

VTGate Advertised MySQL Version

VTGate now advertises MySQL version 8.0.30. This is a breaking change for clients that rely on the VTGate advertised MySQL version and still use MySQL 5.7.
The users can set the mysql_server_version flag to advertise the correct version.

Default MySQL version on Docker

The default major MySQL version used by our vitess/lite:latest image is going from 5.7 to 8.0. Additionally, the patch version of MySQL80 has been upgraded from 8.0.23 to 8.0.30.

Running Vitess on the Operator

If you are using the vitess-operator and want to remain on MySQL 5.7, we invite you to use the vitess/lite:v16.0.0-mysql57 Docker Image.

However, if you are running MySQL 8.0 on the vitess-operator, with for instance vitess/lite:v15.0.2-mysql80, considering that we are bumping the patch version of MySQL 80 from 8.0.23 to 8.0.30, you will have to manually upgrade:

1. Add innodb_fast_shutdown=0 to your extra cnf in your YAML file.
2. Apply this file.
3. Wait for all the pods to be healthy.
4. Then change your YAML file to use the new Docker Images (vitess/lite:v16.0.0, defaults to mysql80).
5. Remove innodb_fast_shutdown=0 from your extra cnf in your YAML file.
6. Apply this file.

vtctld web UI Removal

In v13, the vtctld UI was deprecated. As of this release, the web/vtctld2 directory is deleted and the UI will no longer be included in any Vitess images going forward. All build scripts and the Makefile have been updated to reflect this change.

However, the vtctld HTTP API will remain at {$vtctld_web_port}/api.

vtctld Flag Deprecation & Deletions

With the removal of the vtctld UI, the following vtctld flags have been deprecated:

• --vtctld_show_topology_crud: This was a flag that controlled the display of CRUD topology actions in the vtctld UI. The UI is removed, so this flag is no longer necessary.

The following deprecated flags have also been removed:

• --enable_realtime_stats
• --enable_vtctld_ui
• --web_dir
• --web_dir2
• --workflow_manager_init
• --workflow_manager_use_election
• --workflow_manager_disable

Orchestrator Integration Deletion

Orchestrator integration in vttablet was deprecated in the previous release and is deleted in this release.
Consider using VTOrc instead of Orchestrator.

mysqlctl Flags

The mysqlctl command-line client had some leftover (ignored) server flags after the v15 pflag work. Those unused flags have now been removed. If you are using any of the following flags with mysqlctl in your scripts or other tooling, they will need to be removed prior to upgrading to v16:
--port --grpc_auth_static_client_creds --grpc_compression --grpc_initial_conn_window_size --grpc_initial_window_size --grpc_keepalive_time --grpc_keepalive_timeout

Query Serving Errors

In this release, we are introducing a new way to report errors from Vitess through the query interface.
Errors will now have an error code for each error, which will make it easy to search for more information on the issue.
For instance, the following error:

aggregate functions take a single argument 'count(user_id, name)'

Will be transformed into:

VT03001: aggregate functions take a single argument 'count(user_id, name)'

The error code VT03001 can then be used to search or ask for help and report problems.

If you have code searching for error strings from Vitess, this is a breaking change.
Many error st...

Vitess v15.0.2

Release of Vitess v15.0.2

Major Changes

Upgrade to go1.18.9

Vitess v15.0.2 now runs on go1.18.9.
The patch release of Go, go1.18.9, was one of the main reasons for this release as it includes an important security fixe to net/http package, which is use extensively by Vitess.
Below is a summary of this patch release. You can learn more here.

go1.18.9 (released 2022-12-06) includes security fixes to the net/http and os packages, as well as bug fixes to cgo, the compiler, the runtime, and the crypto/x509 and os/exec packages.

---

The entire changelog for this release can be found here.

The release includes 14 commits (excluding merges)

Thanks to all our contributors: @GuptaManan100, @dbussink, @frouioui, @mattlord, @rsajwani, @shlomi-noach, @vitess-bot[bot]

Vitess v15.0.1

Release of Vitess v15.0.1

Major Changes

Corrupted results for non-full-group-by queries with JOINs

An issue in versions <= v14.0.3 and <= v15.0.0 that generated corrupted results for non-full-group-by queries with a JOIN
is now fixed. The full issue can be found here, and its fix here.

VtAdmin web folder is missing while installing Vitess with local method

When we try to install Vitess locally (https://vitess.io/docs/15.0/get-started/local/#install-vitess) on v15.0, we are getting the following error

npm ERR! enoent ENOENT: no such file or directory, open '/home/web/vtadmin/package.json'

This issue is fixed in 15.0.1. The full issue can be found here, and its fix here.

---

The entire changelog for this release can be found here.

The release includes 25 commits (excluding merges)

Thanks to all our contributors: @GuptaManan100, @frouioui, @harshit-gangal, @rsajwani, @vitess-bot[bot]

Vitess v14.0.4

Release of Vitess v14.0.4

Major Changes

Upgrade to go1.18.7

Vitess v14.0.4 now runs on go1.18.7.
The patch release of Go, go1.18.7, was one of the main reasons for this release as it includes important security fixes to packages used by Vitess.
Below is a summary of this patch release. You can learn more here.

go1.18.7 (released 2022-10-04) includes security fixes to the archive/tar, net/http/httputil, and regexp packages, as well as bug fixes to the compiler, the linker, and the go/types package.

Corrupted results for non-full-group-by queries with JOINs

An issue in versions <= v14.0.3 and <= v15.0.0 that generated corrupted results for non-full-group-by queries with a JOIN
is now fixed. The full issue can be found here, and its fix here.

The entire changelog for this release can be found here.

The release includes 24 commits (excluding merges)

Thanks to all our contributors: @GuptaManan100, @dbussink, @frouioui, @harshit-gangal, @systay, @vitess-bot[bot]

Vitess v15.0.0

Release of Vitess v15.0.0

Summary

• Known Issues
• Breaking Changes
  • Flags
  • VTTablet Flag Deletions
  • Vindex Interface
• Deprecations
  • LogStats Table and Keyspace Deprecated
  • Orchestrator Integration Deprecation
  • Connection Pool Prefill
  • InitShardPrimary Deprecation
• Command-Line Syntax Deprecations
  • VTTablet Startup Flag Deletions
  • VTTablet Startup Flag Deprecations
  • VTBackup Flag Deprecations
• VTGate
  • vtgate --mysql-server-pool-conn-read-buffers
• VDiff2
  • Resume Workflow
• New command line flags and behavior
  • vtctl GetSchema --table-schema-only
  • Support for Additional Compressors and Decompressors During Backup & Restore
  • Independent OLAP and OLTP Transactional Timeouts
  • Support for Specifying Group Information in Calls to VTGate
• Online DDL Changes
  • Concurrent Vitess Migrations
  • VTCtl Command Changes
  • New Syntax
• Tablet Throttler
  • API Changes
• Mysql Compatibility
  • System Settings
  • Lookup Vindexes
• Durability Policy
  • Cross Cell
• New EXPLAIN Format
  • FORMAT=vtexplain
• VTOrc
  • Old UI Removal and Replacement
  • Configuration Refactor and New Flags
  • Example Upgrade
  • Default Configuration Files
• Flags Restructure
  • Flags Diff

Known Issues

• Corrupted results for non-full-group-by queries with JOINs. This can be resolved by using full-group-by queries.

Major Changes

Breaking Changes

Flags

• The deprecated --cpu_profile flag has been removed. Please use the --pprof flag instead.
• The deprecated --mem-profile-rate flag has been removed. Please use --pprof=mem instead.
• The deprecated --mutex-profile-fraction flag has been removed. Please use --pprof=mutex instead.
• The deprecated vtgate/vtexplain/vtcombo flag --planner_version has been removed. Please use --planner-version instead.
• The deprecated flag --master_connect_retry has been removed. Please use --replication_connect_retry instead.
• vtctl commands that take shard names and ranges as positional arguments (e.g. vtctl Reshard ks.workflow -80 -40,40-80) need to have their positional arguments separated from their flag arguments by a double-dash separator to avoid the new parsing library from mistaking them as flags (e.g. vtctl Reshard ks.workflow -- -80 -40,40-80).
• The --cell flag in the vtgate binary no longer has a default value. It is a required argument that has to be specified for the binary to run. Please explicitly specify the flag, if dependent on the flag's default value.
• The --db-config-*-* VTTablet flags were deprecated in v3.0.0. They have now been deleted as part of this release. You must use --db_dba_* now.

vttablet Flag Deletions

The following VTTablet flags were deprecated in 7.0. They have now been deleted

• --queryserver-config-message-conn-pool-size
• --queryserver-config-message-conn-pool-prefill-parallelism
• --client-found-rows-pool-size A different existing flag --queryserver-config-transaction-cap will be used instead
• --transaction_shutdown_grace_period Use --shutdown_grace_period instead
• --queryserver-config-max-dml-rows
• --queryserver-config-allowunsafe-dmls
• --pool-name-prefix
• --enable-autocommit Autocommit is always allowed

Vindex Interface

All the vindex interface methods are changed by adding context.Context as an input parameter.

E.g:

Map(vcursor VCursor, .... ) .... 
	To
Map(ctx context.Context, vcursor VCursor, .... ) ....

This only affects users who have added their own custom vindex implementation.
They are required to change their implementation with these new interface method expectations.

Deprecations

LogStats Table and Keyspace deprecated

Information about which tables are used was being reported through the Keyspace and Table fields on LogStats.
For multi-table queries, this output can be confusing, so we have added TablesUsed, that is a string array, listing all tables and which keyspace they are on.
Keyspace and Table fields are deprecated and will be removed in the v16 release of Vitess.

Orchestrator Integration Deprecation

Orchestrator integration in vttablet has been deprecated. It will continue to work in this release but is liable to be removed in future releases.
Consider using VTOrc instead of Orchestrator as VTOrc goes GA in this release.

Connection Pool Prefill

The connection pool with prefilled connections have been removed. The pool now does lazy connection creation.

InitShardPrimary Deprecation

The vtcltd command InitShardPrimary has been deprecated. Please use PlannedReparentShard instead.

Command-line syntax deprecations

vttablet startup flag deletions

The following VTTablet flags were deprecated in 7.0. They have now been deleted

• --queryserver-config-message-conn-pool-size
• --queryserver-config-message-conn-pool-prefill-parallelism
• --client-found-rows-pool-size --queryserver-config-transaction-cap will be used instead
• --transaction_shutdown_grace_period Use --shutdown_grace_period instead
• --queryserver-config-max-dml-rows
• --queryserver-config-allowunsafe-dmls
• --pool-name-prefix
• --enable-autocommit Autocommit is always allowed

vttablet startup flag deprecations

• --enable-query-plan-field-caching has been deprecated. It will be removed in v16.
• --enable_semi_sync has been deprecated. It will be removed in v16. Instead, set the correct durability policy using SetKeyspaceDurabilityPolicy
• --queryserver-config-pool-prefill-parallelism, --queryserver-config-stream-pool-prefill-parallelism and --queryserver-config-transaction-prefill-parallelism have all been deprecated. They will be removed in v16.
• --backup_storage_hook has been deprecated, consider using one of the builtin compression algorithms or --external-compressor and --external-decompressor instead.

vtbackup flag deprecations

• --backup_storage_hook has been deprecated, consider using one of the builtin compression algorithms or --external-compressor and --external-decompressor instead.

VTGate

vtgate --mysql-server-pool-conn-read-buffers

--mysql-server-pool-conn-read-buffers enables pooling of buffers used to read from incoming
connections, similar to the way pooling happens for write buffers. Defaults to off.

VDiff v2

Resume Workflow

We introduced the ability to resume a VDiff2 workflow:

$ vtctlclient --server=localhost:15999 VDiff --v2 customer.commerce2customer resume 4c664dc2-eba9-11ec-9ef7-920702940ee0
VDiff 4c664dc2-eba9-11ec-9ef7-920702940ee0 resumed on target shards, use show to view progress

$ vtctlclient --server=localhost:15999 VDiff --v2 customer.commerce2customer show last

VDiff Summary for customer.commerce2customer (4c664dc2-eba9-11ec-9ef7-920702940ee0)
State:        completed
RowsCompared: 196
HasMismatch:  false
StartedAt:    2022-06-26 22:44:29
CompletedAt:  2022-06-26 22:44:31

Use "--format=json" for more detailed output.

$ vtctlclient --server=localhost:15999 VDiff --v2 --format=json customer.commerce2customer show last
{
	"Workflow": "commerce2customer",
	"Keyspace": "customer",
	"State": "completed",
	"UUID": "4c664dc2-eba9-11ec-9ef7-920702940ee0",
	"RowsCompared": 196,
	"HasMismatch": false,
	"Shards": "0",
	"StartedAt": "2022-06-26 22:44:29",
	"CompletedAt": "2022-06-26 22:44:31"
}

We leverage this resume capability to automatically restart a VDiff2 workflow that encountered a retryable error.

We also made a number of other enhancements like progress reporting and features needed to make it a full replacement for VDiff v1. You can see more details in the tracking ticket for the VDiff2 feature complete target: #10494

Now that VDiff v2 is feature complete in 15.0, we hope to make it GA in 16.0.

Please see the VDiff2 documentation ...

Vitess v15.0.0-rc1

Release of Vitess v15.0.0-rc1

Summary

• Vindex Interface
• LogStats Table and Keyspace deprecated
• Command-line syntax deprecations
• New command line flags and behavior
• Online DDL changes
• Tablet throttler
• VDiff2
• Mysql Compatibility
• Durability Policy
• New EXPLAIN format
• VTOrc

Known Issues

Major Changes

Breaking Changes

Flags

• The deprecated --cpu_profile flag has been removed. Please use the --pprof flag instead.
• The deprecated --mem-profile-rate flag has been removed. Please use --pprof=mem instead.
• The deprecated --mutex-profile-fraction flag has been removed. Please use --pprof=mutex instead.
• The deprecated vtgate/vtexplain/vtcombo flag --planner_version has been removed. Please use --planner-version instead.
• The deprecated flag --master_connect_retry has been removed. Please use --replication_connect_retry instead.
• vtctl commands that take shard names and ranges as positional arguments (e.g. vtctl Reshard ks.workflow -80 -40,40-80) need to have their positional arguments separated from their flag arguments by a double-dash separator to avoid the new parsing library from mistaking them as flags (e.g. vtctl Reshard ks.workflow -- -80 -40,40-80).

Vindex Interface

All the vindex interface methods are changed by adding context.Context as an input parameter.

E.g:

Map(vcursor VCursor, .... ) .... 
	To
Map(ctx context.Context, vcursor VCursor, .... ) ....

This only affects users who have added their own custom vindex implementation.
They are required to change their implementation with these new interface method expectations.

LogStats Table and Keyspace deprecated

Information about which tables are used was being reported through the Keyspace and Table fields on LogStats.
For multi-table queries, this output can be confusing, so we have added TablesUsed, that is a string array, listing all tables and which keyspace they are on.
Keyspace and Table fields are deprecated and will be removed in the v16 release of Vitess.

Orchestrator Integration Deprecation

Orchestrator integration in vttablet has been deprecated. It will continue to work in this release but is liable to be removed in future releases.
Consider using VTOrc instead of Orchestrator as VTOrc goes GA in this release.

Connection Pool Prefill

The connection pool with prefilled connections have been removed. The pool now does lazy connection creation.
Following flags are deprecated: queryserver-config-pool-prefill-parallelism, queryserver-config-stream-pool-prefill-parallelism, queryserver-config-transaction-prefill-parallelism
and will be removed in future version.

Command-line syntax deprecations

vttablet startup flag deletions

The following VTTablet flags were deprecated in 7.0. They have now been deleted

• --queryserver-config-message-conn-pool-size
• --queryserver-config-message-conn-pool-prefill-parallelism
• --client-found-rows-pool-size --queryserver-config-transaction-cap will be used instead
• --transaction_shutdown_grace_period Use --shutdown_grace_period instead
• --queryserver-config-max-dml-rows
• --queryserver-config-allowunsafe-dmls
• --pool-name-prefix
• --enable-autocommit Autocommit is always allowed

vttablet startup flag deprecations

• --enable-query-plan-field-caching is now deprecated. It will be removed in v16.
• --enable_semi_sync is now deprecated. It will be removed in v16. Instead, set the correct durability policy using SetKeyspaceDurabilityPolicy
• --queryserver-config-pool-prefill-parallelism, --queryserver-config-stream-pool-prefill-parallelism and --queryserver-config-transaction-prefill-parallelism have all been deprecated. They will be removed in v16.

New command line flags and behavior

vtgate --mysql-server-pool-conn-read-buffers

--mysql-server-pool-conn-read-buffers enables pooling of buffers used to read from incoming
connections, similar to the way pooling happens for write buffers. Defaults to off.

VDiff2

We introduced the ability to resume a VDiff2 workflow:

$ vtctlclient --server=localhost:15999 VDiff --v2 customer.commerce2customer resume 4c664dc2-eba9-11ec-9ef7-920702940ee0
VDiff 4c664dc2-eba9-11ec-9ef7-920702940ee0 resumed on target shards, use show to view progress

$ vtctlclient --server=localhost:15999 VDiff --v2 customer.commerce2customer show last

VDiff Summary for customer.commerce2customer (4c664dc2-eba9-11ec-9ef7-920702940ee0)
State:        completed
RowsCompared: 196
HasMismatch:  false
StartedAt:    2022-06-26 22:44:29
CompletedAt:  2022-06-26 22:44:31

Use "--format=json" for more detailed output.

$ vtctlclient --server=localhost:15999 VDiff --v2 --format=json customer.commerce2customer show last
{
	"Workflow": "commerce2customer",
	"Keyspace": "customer",
	"State": "completed",
	"UUID": "4c664dc2-eba9-11ec-9ef7-920702940ee0",
	"RowsCompared": 196,
	"HasMismatch": false,
	"Shards": "0",
	"StartedAt": "2022-06-26 22:44:29",
	"CompletedAt": "2022-06-26 22:44:31"
}

Please see the VDiff2 documentation for additional information.

New command line flags and behavior

vtctl GetSchema --table-schema-only

The new flag --table-schema-only skips column introspection. GetSchema only returns general schema analysis, and specifically it includes the CREATE TABLE|VIEW statement in the schema field.

Support for additional compressors and decompressors during backup & restore

Backup/Restore now allow you many more options for compression and decompression instead of relying on the default compressor(pgzip).
There are some built-in compressors which you can use out-of-the-box. Users will need to evaluate which option works best for their
use-case. Here are the flags that control this feature

• --compression-engine-name
• --external-compressor
• --external-decompressor
• --external-compressor-extension
• --compression-level

--compression-engine-name specifies the engine used for compression. It can have one of the following values

• pargzip (Default)
• pgzip
• lz4
• zstd
• external

where 'external' is set only when using a custom command or tool other than the ones that are already provided.
If you want to use any of the built-in compressors, simply set one of the above values for --compression-engine-name. The value
specified in --compression-engine-name is saved in the backup MANIFEST, which is later read by the restore process to decide which
engine to use for decompression. Default value for engine is 'pgzip'.

If you would like to use a custom command or external tool for compression/decompression then you need to provide the full command with
arguments to the --external-compressor and --external-decompressor flags. --external-compressor-extension flag also needs to be provided
so that compressed files are created with the correct extension. If the external command is not using any of the built-in compression engines
(i-e pgzip, pargzip, lz4 or zstd) then you need to set --compression-engine-name to value 'external'.

Please note that if you want the current production behavior then you don't need to change any of these flags.
You can read more about backup & restore [here] (https://vitess.io/docs/15.0/user-guides/operating-vitess/backup-and-restore/).

If you decided to switch from an external compressor to one of the built-in supported compressors (i-e pgzip, pargzip, lz4 or zstd) at any point
in the future, you will need to do it in two steps.

• step #1, set --external-compressor and --external-compressor-extension flag values to empty and change --compression-engine-name to desired value.
• Step #2, after at least one cycle of backup with new configuration, you can set --external-decompressor flag value to empty.

The reason you cannot change all the values together is because the restore process will then have no way to find out which external decompressor
should be used to process the previous backup. Please make sure you have thought out all possible scenarios for restore before transitioning from one
compression engine to another.

Independent OLAP and OLTP transactional timeouts

--queryserver-config-olap-transaction-timeout specifies the timeout applied
to a transaction created within an OLAP workload. The default value is 30
seconds, but this can be raised, lowered, or set to zero to disable the timeout
altogether.

Until now, while OLAP queries would bypass the query timeout, transactions
created within an OLAP session would be rolled back
--queryserver-config-transaction-timeout seconds after the transaction was
started.

As of now, OLTP and OLAP transaction timeouts can be configured independently of each
other.

The main use case is to run queries spanning a long period of time which
require transactional guarantees such as consistency or atomicity.

Support for specifying group information in calls to VTGate

--grpc-use-effective-groups allows non-SSL callers to specify groups information for a caller.
Until now, you could only specify the caller-id for the security context used to authorize queries.
As of now, you can specify the principal of the caller, and any groups they belong to.

Online DDL changes

Concur...

Vitess v14.0.3

Release of Vitess v14.0.3

Known Issues

• Corrupted results for non-full-group-by queries with JOINs. This can be resolved by using full-group-by queries.

Major Changes

Fix VTOrc Discovery

In patch releases prior to release 14.0.3, if VTOrc is unable to reach the MySQL instance of a vttablet, it was never able to read discover that tablet again.
This problem could be resolved by restarting the VTOrc so that it discovers all the tablets again, but in a kubernetes cluster where the pods are eviced
frequently, this posed a greater challenge, since some pods when evicted and rescheduled on a different node, would sometimes fail to be discovered by VTOrc.
This has problem has been addressed in this patch by the fix #10662.

The entire changelog for this release can be found here.

The release includes 12 commits (excluding merges)

Thanks to all our contributors: @GuptaManan100, @frouioui, @harshit-gangal, @mattlord, @vitess-bot[bot]