Skip to content

Releases: vitessio/vitess

Vitess 6.0.20-20200525 Patch Update

25 May 19:12
5245945
Compare
Choose a tag to compare

This issue addresses a regression in vtgate performance, which was significant enough that it was decided to backport to the release branch. In addition to this bug fix, there remains a known issue where XtraBackup is not compatible with MySQL 8.0.20. Percona is aware of the issue, and there are no known fixes at this point in time.

Bugs Fixed

  • Vtgate: a performance regression was identified as part of the refactoring of vtgate's executor. The executor now maintains a cached copy of the vschema. #6220 #6203

Vitess 6.0.20-20200519 Patch Update

19 May 20:43
f758b52
Compare
Choose a tag to compare

This update fixes a number of issues related to backup discovered in Vitess 6, and is a recommended update for all users. In addition to these bug fixes, there remains a known issue where XtraBackup is not compatible with MySQL 8.0.20. Percona is aware of the issue, and there are no known fixes at this point in time.

Bugs Fixed

  • VTTablet: An issue exists where transaction pool timeout did not apply, and connections never expired #6197
  • VTTablet: An issue was discovered where copying a backup to S3 failed with an error of closed network connection: cannot copy data. This has been worked around by retrying of S3 upload requests. #6199
  • VTTablet: The included version of pgzip has been upgraded to v1.2.4. This helps address an occasional panic that was observed with xtrabackup backups. #6198

Vitess 6.0.20-20200508 Patch Update

08 May 15:19
147bc5a
Compare
Choose a tag to compare

Known Issues

  1. While running a builtin backup with large databases, you might sometimes get this error: use of closed network connection: cannot copy data: MultipartUpload: upload multipart failed.
    We believe this is caused by the newer aws-sdk-go version we upgraded to in 5.0. It might be this problem filed upstream.
  2. On xtrabackup backed up tablets on the new build, you might occasionally hit this panic:
    Rendezvous of RPC that terminated with (StatusCode.UNKNOWN, caught panic during Backup: bufio: writer returned negative count from Write This also seems to be related to aws-sdk-go, though we haven't traced it to a known issue.
  3. There is a known regression where the transaction pool timeout will not apply, and connections never expire. We are working on a fix for this issue, and will release an update shortly.

Release Notes

Two regressions were discovered in Vitess 6 after release. While neither are critical, they are visible enough that we wanted to release a patch update:

  • A regression in the packaging of tar.gz files moved the examples from examples/ to share/vitess/examples. This made the install from binary instructions incorrect. #6159 #6160

  • A regression in how select database() returned the current database in use broke how the MySQL command-line client changes the currently selected database. This issue is specific to the mysql client, and is not known to affect application usage of com_init_db or use <dbname> #6157 #6158

Vitess 6 General Availability

29 Apr 14:18
Compare
Choose a tag to compare

Known Issues

  1. While running a builtin backup with large databases, you might sometimes get this error: use of closed network connection: cannot copy data: MultipartUpload: upload multipart failed.
    We believe this is caused by the newer aws-sdk-go version we upgraded to in 5.0. It might be this problem filed upstream.
  2. On xtrabackup backed up tablets on the new build, you might occasionally hit this panic:
    Rendezvous of RPC that terminated with (StatusCode.UNKNOWN, caught panic during Backup: bufio: writer returned negative count from Write. This also seems to be related to aws-sdk-go, though we haven't traced it to a known issue.
  3. There is a known regression where the transaction pool timeout will not apply, and connections never expire. We are working on a fix for this issue, and will release an update shortly.

Incompatible Changes

This release includes the following changes, which may result in incompatibilities when upgrading from a previous release:

Changes to Local Examples

The local examples have been updated to use the new VReplication workflows for MoveTables and Resharding. This obsoletes the previous workflows of Vertical Split Clone and Horizontal Sharding. The earlier examples remain available in the “local.legacy” example directory for now. #6016 #6015 #6052 #6072 #6106 #6121

Change to Helm Chart

The helm chart has been updated to default to using Kubernetes as the Topology Server provider. This change was required to remove the dependency on etcd-operator, which is no longer under active support. #5989

Earlier Kubernetes Operator Retired

Support for the Vitess Kubernetes Operator has been removed due to a lack of maintainers. Adoption is understood to be low, but users are encouraged to migrate to using the Helm charts which continue to be supported, or PlanetScale’s Vitess Operator (provided by a third party) #5873

SQL Support: Keywords are no longer lowercase

Vitess no longer lower cases SQL keywords. In corner cases, this may affect applications which made specific use of SQL keywords, and then referred to a column in the result set #5897

SQL Support: Parenthesis are now removed from the AST

Vitess now simplifies expressions to remove parentheses from expressions. In corner cases, this may affect applications that use an expression that contains parens and then refer to it as a column in the result. #5906

Statement Based Replication, SplitQuery and UpdateStream

Code for the unsupported features of Statement-based replication (SBR), SplitQuery and UpdateStream has now been removed. Users using SBR are encouraged to switch to using Row-based replication instead. #5940 #5972 #5928

Messaging: First Class Citizen + API Improvements

The messaging feature has had several breaking changes and feature additions

  • Direct inserts are now supported: INSERT INTO msg_table SELECT col1, col2...
  • time_scheduled and time_created are no longer required nor automatically populated.
  • priority is a new required field. Messages with a lower priority will be processed first.
  • Message retention is now measured after time_acked instead of time_scheduled
  • New table options vt_min_backoff and vt_max_backoff set bounds on exponential backoff for message retries
  • 33% backoff jitter has been added for messages that are getting postponed to prevent thundering herds.

Users of the messages feature are encouraged to read through the full details in issue #5947 #5948 #6114

RPM and DEB Packages

Vitess binaries and examples are now installed in system standard paths, rather than the /vt directory #5836

Vitess Go Driver

Values of type sqltypes.Timestamp are now automatically converted to time.Time values
#5788

Simplified VTtablet Schema Load Process

The process of loading vttablet schema has been simplified, as many of the statistics loaded were expensive to gather but remained unused. This may lead to compatibility breakage in the use of the features: Sequences, VStreamer and Messages #5951

Planned Reparent

The implementation of Planned Reparent and Emergency Reparent have been improved, with more graceful handling of failure modes. While this change is functionality neutral, it is important that Vitess components are upgraded in the recommended order. #6050

Bugs Fixed

  • Binlog Client: Vitess components now randomize the server_id when acting as a MySQL replica. #5998

  • heartbeat: Prevent a scenario where tables could be created on a replica leading to errant transactions. Now tables are only created on the master with binlog ON, using the app_user not dba_user #6053

  • Messages: For the mysql protocol, buffered results coming from vitess are flushed a minimum of every 100ms, and configurable by mysql_server_flush_delay. This helps prevent against a case where intermittent streaming could delay messages from being sent. #5879

  • mysqlctl: Correctly escape and quote passwords that contain special characters. #5761

  • VReplication: Several improvements to failure handling and miscellaneous fixes #5753 #5792 #5801 #5842 #5935 #5905 #5845 #5942

  • Vtcompose: Add a check to make sure that files can be read #6024

  • Vtcompose: refactor vtcompose for improved testability #5994

  • Vindexes: Support for a region_json Vindex #6017 #5806 #5955

  • Vtctl: Before applying a VSchema, vtctl now correctly checks that the schema exists. #5931

  • Vtctl: Vtctl now exits early if the topo_global_root was set incorrectly #6119

  • Vtctl: EnsureVSchema and GetOrCreateShard now pass in the cell name. This helps prevent against corruption, which could occur from auto-rebuilding of cells automatically when creating a new shard or keyspace. #5930

  • VTExplain: An issue where incorrectly formatted comments could lead to a hang has been fixed. #5984

  • VTGate: Leading zeros were incorrectly dropped in microsecond timestamps when using prepared statements #5901

  • VTGate: Setting the transaction_mode in a mysql session is now case insensitive. #5963

  • VTGate: When updating consistent indexes, VTGate applies an optimization where it checks if anything really changed before applying an update. In the case that this check results in errors, VTGate will now skip the optimization and execute the update regardless. #5883

  • VTGate: Remove allocation of unused bytes #5874

  • VTGate: The gateway interface functionality has been simplified in anticipation of two new gateway implementations: TabletGateway and VTDirectGateway #5978 #5992

  • VTGate: The process of watching for VSchema updates from the topology server now recovers successfully if it receives an invalid update. This helps prevent against a scenario where a malformed vschema could take down all vtgate servers. #6051

  • VTGate: Previously VTGate performed health checks against all keyspaces, not just keyspaces listed in keyspaces_to_watch. Healthchecks are now limited correctly. #5815

  • VTGate: SQL Support: database() is now only rewritten against dual #5793

  • VTGate: Plan Cache is now disabled for statements with nil instructions #6045

  • VTGate: Make sure to only accept DDL that can be parsed, at least partially #6023

  • VTGate: Log when accepting a new connection fails #5881

  • VTGate: Prepared Statements did not correctly handle signed integers #5974

  • VTGate: Executor statistics did not correctly account for dividing by zero, leading to a potential panic. #5904

  • VTGate: The error message for unsupported DDL creation has been improved #6109

  • VTGate: The metric QueriesProcessedByTable was not correctly incremented with streaming queries processed by vtgate #6057

  • GRPC Client: The default keep alive and keep alive timeout values have been changed from unlimited to 10 seconds. This helps prevent cases where queries could hang while waiting for a failed pod or node. In addition, the defaultMaxMessageSize is now set to 16MB #5922 #3890

  • VTTablet: local_metadata is now populated after an agent has been initialized. This helps prevent against the ambiguous case where both -init_populate_metadata and -restore_from_backup are specified. #5783

  • VTTablet: A hang could occur when shutting down a tablet server when the Topo Server is unavailable. Tablet servers will now time out correctly. #5865

  • VTTablet Health Check: A warning is now logged when the vttablet StreamHealth RPC fails #5917

  • VTTablet Planned Reparent: A new context is now created for calling UndoDemoteMaster. This helps prevent a scenario where the existing context may have expired. #5841

  • VTTablet: A panic could occur when calling NewDBConnNoPool #6086

  • VTTablet: Queries of type STREAM_EXECUTE were inadvertently dropped from logging #6056

  • Zookeeper Topo: Trailing newlines are now ignored when specifying -topo_zk_auth_file
    #6030

  • Misc: Stats counters would erroneously print warnings to log files when adding a negative value. This warning did not consider the case where counters are reused. #5988

  • Misc: Binary log events with the special name of "mysql.rds_*" are now skipped. This helps improve compatibility with Amazon RDS flavors #5966

Functionality Added or Changed

  • Docker: sysstat and strace are now bundled in vitess/lite images #5814

  • Docker: new lite images based on Red Hat UBI 7 are now available #6100

  • Example Code: An are you alive example has been added #5646

  • Example Code: The demo application has been migrated from Python to Go #6022

  • Example Code: A set of notes and scripts on how to use Vitess with common frameworks #6034

  • Helm charts: now work with the latest Kubernetes version. An issue where the 308_...

Read more

Vitess 5 General Availability

04 Feb 16:15
Compare
Choose a tag to compare

Incompatible Changes

This release includes the following changes, which may result in incompatibilities when upgrading from a previous release:

Deprecated Support for Vagrant
Support for deployment on Vagrant is deprecated, and will be removed in the next release of Vitess. The recommended upgrade path is local deployment on macOS.

Deprecation of vtgate option disable_local_gateway
The option disable_local_gateway is parsed but ignored by vtgate. It is now deprecated, and will be removed in a future version #5741

Web_dir2 and web_dir options for vtctld
The options web_dir2 and web_dir are now parsed but ignored. This is a behavior change from 5.0 beta, where the options were removed. We intend to remove these options again in Vitess 6.0, but discovered compatibility issues with existing tooling. #5744 #5747

Bugs Fixed

  • Consul: The bundled consul version has been bumped to 1.5.1, addressing CVE-2019-12291 and CVE-2018-19653.
  • VReplication: Fixed a panic when connecting to an external MySQL server #5753
  • Vttablet: Address #5734 by forming MySQL error packet on query parse error during
  • Vtgate: Fixed an issue with prepared statements returning a syntax error #5735
  • Mysqlctl: Fixed a regression where the option mysqlctl_mycnf_template was accidentally removed during refactoring. #5717
  • Mysqlctl: Fix a regression where refactoring configuration files removed the options relay-log and relay-log-index from included configuration #5745
  • Vttablet: Fixed a problem with vttablet consuming too much memory during a restore from backup when running with option -s3_backup_log_level LogDebugWithRequestRetries #5777

Functionality Added or Changed

  • Vtctld: The reference to web/vtctld has been removed from the Makefile #5778
  • mysqlctld: An option has been added to mysqlctl to allow passdown of LD_PRELOAD libraries. This is useful in the case where a custom memory allocation library needs to be specified. #5730
  • Logging: Add datadog plugin for tracing #5681
  • Add support for DISTINCTROW as an alias for DISTINCT SQL syntax #5726 #5718
  • Vstream: support default values #5736
  • VReplication: Add support for ExternalizeVindex #5716
  • ApplySchema: Add support for ALTER DATABASE #5733

Build Environment Changes

Vitess 5.0.1 Beta

15 Jan 16:34
5165f85
Compare
Choose a tag to compare
Vitess 5.0.1 Beta Pre-release
Pre-release

Incompatible Changes

This release fixes an issue discovered after releasing Vitess 5.0.0 Beta (#5711). We recommend reading the release notes for 5.0.0 when upgrading from an earlier release.

Bugs Fixed

  • mysqlctl: Vitess programs would not run if configuration files could not be located #5711
  • VReplication: The keyrange was incorrectly being quoted twice #5715
  • Release packaging: Fix issues discovered in automated release scripts #5714
  • Docker images: provide a workaround for missing configuration files in vitess/lite image #5707
  • Change http links to https #5697
  • helm: Fix examples to remove undefined flag #5696

Functionality Added or Changed

  • VReplication: CreateLookupVindex command with backfill #5692

Vitess 5.0.0 Beta

13 Jan 22:01
1b384b8
Compare
Choose a tag to compare
Vitess 5.0.0 Beta Pre-release
Pre-release

Known Issues

Important! This release contains a bug, where MySQL configuration files were intended to be bundled in the mysqlctl binary, are not correctly bundled. The workaround is to ensure that $VTROOT/config is available on your system. We intend to fix this issue before final release of Vitess 5.0.

Incompatible Changes

This release includes the following changes, which may result in incompatibilities when upgrading from a previous release:

Removal of binaries from release packages
The release packages have been improved to no longer ship unneeded dependencies. If you previously installed etcd, zookeeper or consul as part of the Vitess package, you will now need to download and install it separately. #5545 #5645

Refactoring MySQL Configuration Files
The included my.cnf files for MySQL have been simplified and improved. There may be some isolated cases where the difference in configuration may lead to compatibility issues with a previous installation of MySQL. #5326 #5517 #5704

VTROOT, VT_MYSQL_ROOT, VTDATAROOT changes
You are no longer required to specify the environment variables VTROOT, VT_MYSQL_ROOT, VTDATAROOT in order to run the Vitess examples. These variables have been changed to autodetect. Use of these variables is still permitted, but discouraged, as they may be removed in a future release. #5488 #5506 #5654 #5560

Elimination of VTTOP environment variable
The environment variable VTTOP has been eliminated, with all uses changed to VTROOT instead. In most cases VTROOT can also be autodetected, so it is encouraged to remove VTTOP from any scripts. #5527 #5632 #5546 #5548

Removal of vtqueryserver
The unused vtqueryserver utility has been removed. #5583

web_dir2 and web_dir options removed from vtctld
The web files used by vtctld are now embedded into the vtctld binary, simplifying configuration. This results in a behavior change since web_dir and web_dir2 are no longer valid parameters when starting vtctld. Please be sure to update your startup scripts accordingly. #5597

Examples now check for dependencies
The examples now check for shell programs that are required to execute correctly. If you do not have the following programs in your PATH, it will result in an error: mysqld etcd etcdctl curl vtctlclient vttablet vtgate vtctld mysqlctl #5543

Changes to build from source
The build from source instructions have changed. See https://vitess.io/docs/contributing/build-from-source/ for details.

Bugs Fixed

  • The stability of prepared statements has been improved #5562 #5441
  • The portability of shell scripts has been improved. #5384 #5480
  • The semi-sync extension is now optional #5483
  • A panic could occur when a sequence name was bad. #5621
  • Loading TLS and x509 certificates now consumes less memory. #5423
  • Helm no longer complains about nil values in templates. #5484
  • Add workaround for common MySQL replication error #5627
  • Add null as a valid constant SELECT value for vtexplain #5624
  • Creating new shards could overwhelm a topo server. Vitess now no longer performs actions concurrently when creating shards in the topo. #5436
  • VTExplain could race when waiting around for the fakesqldb tabletserver #5476
  • VTGate now correctly resolves columns in information_schema tables. #5608
  • The query SHOW FULL COLUMNS now works correctly with a filter #5558
  • Backups now pass the correct filesize to Ceph #5369
  • Vtbackup will now stop replication after reaching the goal position. #5437
  • The bundled version of log4j has been upgraded #5691
  • VTTablet now accepts external tablets in Azure MySQL #5683
  • VTGate now advertises protocol support for FOUND ROWS #5676
  • A buffered reader is now placed in front of xtrabackup’s STDOUT pipe, potentially reducing memory usage #5666
  • Consultopo: Enforce a connection heartbeat for Watch() #5673
  • The tabletmanager now ensures that it is fully shutdown before closing the topology client #5667

Functionality Added or Changed

  • Several improvements to VReplication #5432 #5628 #5367 #5594 #5575 #5574 #5509 #5500 #5498 #5494 #5491 #5457 #5456 #5454 #5429 #5490 #5515 #5689 #5688 #5669
    Including:
    • Support for file+pos based replication, allowing replication in from MySQL instances which do not have GTIDs enabled
    • New Materialize command
    • Support for Vdiff, Reshard
    • Vindex and multi-column support
  • Several Improvements to Docker files. #5592 #5510 #5470 #5458 #5444 #5492 #5661
    Including:
    • A refactoring of vitess/lite files
    • Support for aarch64
    • The default mysql data files are now removed from the vitess/lite images
    • The latest xtrabackup is always retrieved
    • Jemalloc and tcmalloc are now bundled.
  • Mysqlctl now checks that it is not being executed as root. This helps debugging scenarios, since mysqld will refuse to execute as root by default. #5419 #5644
  • The examples have been improved. Improvements include a check that all dependencies are installed, and warn if it looks like selinux or apparmor prevented mysqlctl from starting #5543 #5573 #5461 #5433 #5426 #5386
  • The bootstrap process will search for etcd in the PATH, and won’t install it if it is present. In addition, etcd on aarch64 is now supported. #5427 #5481 #5428 #5495 #5434 #5425
  • VTGate now supports proxy protocol #5563
  • There is now an option for TLS enabled etcd server connections. #5496
  • Vtgate now advertises itself as a MySQL 5.7 server #5482
  • A vt_monitoring user has been added to init_db.sql. #5486
  • Mysqlctl now supports bootstrapping MariaDB 10.4. Please note that MariaDB 10.4 is not yet supported. #5374
  • VTGate now provides a scatter of query statistics #5422
  • Vttablet now provides better granularity for errors that come out of the transaction pool #5373
  • Experimental support for region-aware vindexes, and xxhash vindexes #5390 #5439
  • VTTablet servers now support an option to disable caching field names to avoid schema mismatches #5572
  • VTGate: Query routing has been improved when using LEFT JOIN queries #5551
  • VTGate: The parser now supports additional types of BEGIN statements #5550
  • VTTablet servers no longer rechecks the mysql port if it was explicitly provided. This helps ensure operation in proxied environments. #5383
  • The 64 bit hash implementation has been changed from 3DES to DES #5529
  • Demote a master vttablet to a SPARE instead of REPLICA is now supported #5520
  • Vttestserver now supports vschema operations via vtgate #5582
  • The vtgate resolver has been simplified #5679

Build Environment Changes

Vitess 4.0.1

15 Nov 00:24
0629f0d
Compare
Choose a tag to compare

An issue was discovered in the previous packages, where packages reported the wrong version.

Bugs Fixed

  • Add stronger protection against running examples as root #5431

Vitess 4.0

05 Nov 15:46
cc07de2
Compare
Choose a tag to compare

Important Changes

  • Vitess 4.0 no longer supports Statement-Based Replication. Users upgrading will need to switch to Row-Based Replication.
  • V2 Routing is now deprecated and will be removed in a future release. This was an undocumented API that was never officially supported.
  • The default topology server has switched to etcd. ZooKeeper and Consul remain fully supported.
  • Support for InfluxDB as a monitoring backend has been removed. Prometheus is recommended in place.
  • VReplication support is experimental. We are satisfied with the stability and test coverage of this feature but may make changes to the command-line interface based on feedback received. These changes will not follow the Vitess deprecation policy.
  • Prepared statements support is experimental. Support for prepared statements was merged later in the development cycle, and anticipate that there may still be undiscovered edge-cases.
  • The internal representation of topology has been altered between Vitess 3.0 and 4.0. For the most part, it is backward and forwards compatible. The only caveat is to not upgrade to 4.0 while performing horizontal/vertical resharding.
  • The internal Vitess tables VReplication, local_metadata and shard_metadata tables have been modified to support multiple databases in the same mysql instance. This schema change is expected to be forwards and backwards compatible with Vitess 3.0.
  • Support for VStream is experimental. We invite feedback from users before stabilizing this feature in an upcoming release.
  • The included MySQL configuration files bundled with Vitess have been improved. This may lead to cases where a MySQL server started with Vitess 3.0 will not start in Vitess 4.0. Please see the error log for the MySQL server for details.
  • The configuration flags timeout, replication_timeout, gcs_backup_storage_project and lock_timeout have been deprecated and will be removed in the next release of Vitess.
  • The reparenting process has been significantly improved in Vitess 4.0. It is important that you follow the recommended upgrade order for components so that reparents work properly while components are on mixed versions.

Bugs Fixed

Functionality Added or Changed

Build Environment Changes

Read more

Vitess 3.0

10 Dec 04:47
4f192d1
Compare
Choose a tag to compare

Vitess 3.0 many new features over 2.2.

Download

  • Server binaries are available in the Docker image "vitess/lite:v3.0.0-rc.1".
  • Java client packages are published on Maven Central (To be uploaded).
  • Source code is available via the tag "v3.0.0-rc.1".

Upgrading

All Vitess 3.0 changes are backward compatible with the 2.2 release. Please make sure that you follow the recommended upgrade order: vttablet->vtctld->vtgate.

Changes since 2.2

Usability

  • Tutorial that runs on Minikube
  • Tutorial that runs on local host
  • Simplified db parameters for vttablet
  • Formal support for externally managed mysql

Monitoring

  • Prometheus plug-in

SQL constructs

  • Comment directives: QUERY_TIMEOUT_MS=1000, MULTI_SHARD_AUTOCOMMIT, SKIP_QUERY_PLAN_CACHE
  • Partial Scatter
  • New ALTER VSCHEMA syntax
  • Multi-table DDL statements
  • Workbench connectivity
  • Cross-shard subqueries
  • Improved handling of select * constructs

Performance

  • vtbench: Benchmarking tool
  • MySQL protocol performance improvements
  • Faster reparents
  • Parallelized backups (4X faster)
  • Snappier InitShardMaster and PlannedReparentShard
  • Improved coordination with Orchestrator

New MySQL versions support

  • MySQL 8.0 and MariaDB 10.3

Build

  • Pure Go 1.11. You can now build with CGO_ENABLED=0.

Resharding and VReplication

  • ShowResharding
  • CancelResharding
  • Safer MigrateServedTypes
  • Auto-reverse direction of MigrateServedTypes to allow rollback of a previous MigrateServedTypes
  • Workflow SplitDiff parallelized across shards
  • SplitClone uses destination master for comparison