This repository has been archived by the owner on Dec 16, 2022. It is now read-only.
Vitess v8.0 Release branch #194
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Alkin Tezuysal <alkin.tezuysal@gmail.com>
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
7.0.2 Release Notes
pitr: fix deadlock in ff code
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Signed-off-by: deepthi <deepthi@planetscale.com>
Signed-off-by: Andrew Mason <amason@slack-corp.com>
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Add GenerateShardRanges to vtctl commands
Signed-off-by: deepthi <deepthi@planetscale.com>
…re-deleted-tables Materialization: Do not error if tables that are not part of the stream go missing
add myself to CODEOWNERS
Signed-off-by: Andrew Mason <amason@slack-corp.com>
Signed-off-by: Andrew Mason <amason@slack-corp.com>
Signed-off-by: Andrew Mason <amason@slack-corp.com>
Zero autoinc mode
VStreamer: handle SET statements in the binlog stream
Signed-off-by: Andrew Mason <amason@slack-corp.com>
Signed-off-by: Harshit Gangal <harshit@planetscale.com>
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Fix bug where password was never passed. Signed-off-by: Jacques Grove <aquarapid@gmail.com>
Signed-off-by: Jacques Grove <aquarapid@gmail.com>
Signed-off-by: GuptaManan100 <manan@planetscale.com>
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
fixed orchestrator govet errors
Signed-off-by: GuptaManan100 <manan@planetscale.com>
sort tabledefinitions before comparing them
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
[8.0] Fix error handling in olap mode
Signed-off-by: Alkin Tezuysal <alkin.tezuysal@gmail.com>
Signed-off-by: Alkin Tezuysal <alkin.tezuysal@gmail.com>
…-lite [8.0] docker: pin mysql 5.7 and 8.0 versions for xtrabackup compatibility
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
VDiff/Tablet Picker: fix picker retry logic to handle canceled context
Signed-off-by: Alkin Tezuysal <alkin.tezuysal@gmail.com>
Signed-off-by: Alkin Tezuysal <alkin.tezuysal@gmail.com>
Signed-off-by: Alkin Tezuysal <alkin.tezuysal@gmail.com>
8_0_0 Release Notes
… writing an error packet Signed-off-by: GuptaManan100 <manan@planetscale.com>
…cket [8.0] Bug which caused the connection to not close in case of error writing an error packet
we can't group together multiple varchar values into a IN query, because that forces use to compare varchar values on the vtgate level which we can't do correctly yet Signed-off-by: Andres Taylor <andres@planetscale.com>
[8.0] don't try to compare varchars in vtgate
Fixes vitessio#6988 Signed-off-by: David Weitzman <dweitzman@pinterest.com>
8.0 backport: Correctly initialize the TabletType stats
Patch to handle this case while we move towards a permanent fix upstream. cf. vitessio#4661 vitessio#4669
Signed-off-by: Michael Demmer <mdemmer@slack-corp.com>
|
The process described here makes sense to me. What a merge / git hell 😓 |
ajm188
reviewed
Mar 5, 2021
There was a problem hiding this comment.
A couple inline comments, and also
Concerns:
- Emergency Reparent Refactor vitessio/vitess#6449 - still has segfault problem if doing an ERS on a shard that has a tablet falsely claiming MASTER status. This is covered in [reparentutil / ERS] confirm at least one replica succeeded to
SetMaster, or fail vitessio/vitess#7486, but incorporating that is a little tricky. Options:
a. expand cherry-picks to includereparentutiland the corresponding bugfix therein
b. backport equivalent fix (don't attempt to wait-for-relay-logs-to-apply on any candidate not instatusMap(because it's actually inmasterStatusMap)) - a
Notes:
- I ignored everything under
go/vt/orchestrator/,go/test/,go/vt/sqlparser,go/vt/vtgate/planbuilder/,go/vt/vttablet/tabletserver/planbuilder,helm,java/**/,test/**,web/** - There's another gRPC change I called out inline around changing the default keepalive timeout.
| # See: https://github.com/mattn/go-sqlite3/issues/803 | ||
| # Work around by dropping optimization level from default -O2. | ||
| # Safe, since this code isn't performance critical. | ||
| export CGO_CFLAGS := -O1 |
There was a problem hiding this comment.
Maybe, i don't actually know off hand where the CGO is utilized; possibly @rafael has thoughts? OP PR is not super informative vitessio#6632
| github.com/smartystreets/goconvey v1.6.4 // indirect | ||
| github.com/spf13/cobra v0.0.5 |
There was a problem hiding this comment.
This seems wrong to me... are there no new vtctldclient PRs in here?
There was a problem hiding this comment.
Don't believe so -- vitessio#6752 is the only vtctld change in this branch is the one listed as 8.0 in your spreadsheet. This dependency was removed as part of vitessio#6547
I make no claims to correctness only accuracy vs upstream 😓 https://github.com/vitessio/vitess/blob/release-8.0/go.mod
I will note that it does get correctly added back as part of the vtctld branch #195
There was a problem hiding this comment.
cool, this is good to me. my expectation is this will all resolve when we finally catch up to the vtctld work in an official upstream release
| err := sConn.handleNextCommand(&handler) | ||
| if err != nil { | ||
| t.Fatalf("error handling command: %v", err) | ||
| kontinue := sConn.handleNextCommand(&handler) |
| @@ -86,7 +86,7 @@ var ( | |||
|
|
|||
| // EnforcementPolicy MinTime that sets the keepalive enforcement policy on the server. | |||
| // This is the minimum amount of time a client should wait before sending a keepalive ping. | |||
| GRPCKeepAliveEnforcementPolicyMinTime = flag.Duration("grpc_server_keepalive_enforcement_policy_min_time", 5*time.Minute, "gRPC server minimum keepalive time") | |||
| GRPCKeepAliveEnforcementPolicyMinTime = flag.Duration("grpc_server_keepalive_enforcement_policy_min_time", 10*time.Second, "gRPC server minimum keepalive time") | |||
There was a problem hiding this comment.
This is another important callout unless we're explicitly setting a value for this everywhere.
There was a problem hiding this comment.
Yup - we explicitly set this to 30s
There was a problem hiding this comment.
Well, i guess we explicitly set it for tablets; we should give our vtgate GRPC clients a heads up (or decide upon + set some explicit value) for vtgates
Backport some panic protection during ERS This takes the core of the change from vitessio#7486 and backports it into 8.0. Signed-off-by: Richard Bailey <rbailey@slack-corp.com>
OP vitessio#7523 This basically protects from trying to catch up on replication on hosts that are likely not replicating. Signed-off-by: Richard Bailey <rbailey@slack-corp.com>
Adds better error when there are timeouts in resource pools
ajm188
approved these changes
Mar 10, 2021
This checks if a vtgate is currently filtering keyspaces before requesting the TopoServer. This is necessary because a TopoServer can't be accessed in those cases as the filtered Topo in those cases could make it unsafe to make writes since all reads would be returning a subset of the actual topo data. The only use of the requested topoServer that I found was in the DDL handling path and was introduced in vitessio#6547. This is deployed on dev but should get testing (endtoend or unit, unclear on best path atm) before going upstream.
ajm188
approved these changes
Mar 15, 2021
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Upstream Release Notes
Incompatible Changes
The following PRs made changes to behavior that clients might be relying on. They should be reviewed carefully so that client code can be changed in concert with a Vitess release deployment.
The three errors being changed:
A Note about VTctld builds
This does not include everything for our bleeding edge deployments of vtctld to support the feature work we're doing that. That will be rolled out is a different branch that will initially be designed to merge into these changes and going forward (hopefully) easily cherry pick into 9.0 etc.
Slack changes
SetMaster, or fail vitessio/vitess#7486 - 220e4dbvitessio#6630 - Change GRPC message limit SQL error type
vitessio#6628 - Docker - allow BUILD_NUMBER variable override
vitessio#6481 - Update user-facing terminology in the vtctld2 UI
vitessio#6496 - Fixes error log
vitessio#6705 - vtexplain2
vitessio#6752 - Add GenerateShardRanges to vtctl commands
vitessio#6750 - Add support for
SHOW (DATABASES|VITESS_SHARDS|VITESS_TABLETS) LIKEvitessio#6800 - Update build errors in ./go/vt/vtgate for go1.15
Non Slack changes (interesting)
vitessio#6943 - [8.0] docker: pin mysql 5.7 and 8.0 versions for xtrabackup compatibility
vitessio#6449 - Emergency Reparent Refactor
vitessio#6620 - select query on lookup table to lock the row — The select query on a lookup vindex will take exclusive lock if it is a dml query and is executed in a transaction.
vitessio#6762 - tm: call setReadOnly inside ChangeTabletType
vitessio#6818 - Refactor conn.go — substantial change to mysql/conn handling, few docs
vitessio#6515 - vttablet: Open and Close healthStreamer
vitessio#6625 - tm: fix replmanager deadlock
vitessio#6748 - Fix reserved connection in autocommit mode on DML
vitessio#6731 - tx limiter: fix quota leak if Begin() errors or if the tx killer runs
vitessio#6486 - Adds more system settings
vitessio#6493 - Allow enumeration in set system variables
vitessio#6499 - Make sure lookup vindexes are queryable inside transaction
vitessio#6511 - Convert syslog events to normal log events
vitessio#6490 - vttablet: create database if not present
VRep / Workflow related
vitessio#6635 - Update VReplication Timestamp w/ Heartbeat
vitessio#6519 - VReplication and vstreamer metrics
vitessio#6787 - vrepl metrics: add metrics around vstreamer and vreplication metrics
vitessio#6814 - Add validations and logs to fix common issues faced during MoveTables/Reshard
vitessio#6778 - Materialization: Do not error if tables that are not part of the stream go missing
vitessio#6684 - VDiff logs: additional logging around vdiff/vstreamer for observability
vitessio#6669 - VRep: improve few error/log messages
vitessio#6533 - Workflow: Add List-All Command
vitessio#6477 - add skip flag that can skip comparing source & destination schema when run splitdiff
Misc Neat Stuff to examine
vitessio#6811 - Support for SELECT INTO OUTFILE S3
vitessio#6719 - Managed DROP TABLE
vitessio#6547 - Experimental: automated, scheduled, dependency free online DDL via gh-ost/pt-online-schema-change
vitessio#6634 - heartbeat: lag histogram
vitessio#6543 - Named columns and convenience SQL query interface
vitessio#6691 - Allow switching between OLAP and OLTP
vitessio#6655 - make offset work in OLAP mode
vitessio#6815 - vttablet throttler feature flag: -enable-lag-throttler
vitessio#6668 - vttablet throttling
Non Slack changes (less interesting)
vitessio#7391 - 8.0 backport: Correctly initialize the TabletType stats
vitessio#7270 - [8.0] don't try to compare varchars in vtgate
vitessio#6976 - [8.0] Bug which caused the connection to not close in case of error writing an error packet
vitessio#6790 - 8_0_0 Release Notes
vitessio#6954 - VDiff/Tablet Picker: fix picker retry logic to handle canceled context
vitessio#6943 - [8.0] docker: pin mysql 5.7 and 8.0 versions for xtrabackup compatibility
vitessio#6940 - [8.0] Fix error handling in olap mode
vitessio#6927 - Backport vitessio#6845 - VReplication: Set time zone to UTC while streaming rows
vitessio#6929 - Revert "Docker - upgrade to Debian Buster (release-8.0)"
vitessio#6925 - Backport 6880 - VDiff: make enums comparable
vitessio#6922 - [8.0] healthcheck: use isIncluded correctly to fix replica/rdonly routing bug
vitessio#6909 - [8.0] OnlineDDL bugfix: make sure schema is applied on tablet
vitessio#6906 - [8.0] [Java] Update version to 8.0.0 for release
vitessio#6921 - Backport pull request vitessio#6849 (add timeout for mysqld_shutdown) to 8.0
vitessio#6899 - 8.0 Bugfix: fix parsing of online-ddl command line options
vitessio#6889 - Fixes for flaky tests
vitessio#6858 - [8.0] healthcheck: should receive healthcheck updates from all tablets in cells_to_watch
vitessio#6888 - Docker - upgrade to Debian Buster (release-8.0)
vitessio#6881 - INFORMATION_SCHEMA.TABLE_SCHEMA comparison with dynamic value bugfix
vitessio#6859 - [8.0] Disable reserved connections by default
vitessio#6867 - download zookeeper 3.4.14 from archive site
vitessio#6837 - online-DDL: migration uses low priority throttling
vitessio#6807 - Added support for SHOW TABLE STATUS queries.
vitessio#6828 - Rename Vitess fork of orchestrator to vtorc.
vitessio#6773 - helm/docker: fix Docker builds + tag 7.0.2
vitessio#6821 - Add clusters_to_watch flag to orchestrator. Defaults to all clusters.
vitessio#6817 - Fix and clarify some help options
vitessio#6820 - Add Alkin to maintainers
vitessio#6791 - Mask database name
vitessio#6812 - orc tests: don't set DbName for tests that don't need it.
vitessio#6808 - Fix error around breaking of multistatements
vitessio#6801 - orc tests: add more cases
vitessio#6802 - Add optional ignore to delete statement
vitessio#6775 - Add flags to allow PITR binlog client TLS support
vitessio#6794 - Fix dbname that needs escaping that we broke in fce4cfd
vitessio#6780 - orc: use contexts with timeout for remote operations
vitessio#6770 - decider: endtoend test infrastructure + tests
vitessio#6729 - Strings to enum
vitessio#6786 - tm: change how SetReadOnly is called to avoid errors from externally managed tablets
vitessio#6785 - Add myself (Rohit) to CODEOWNERS
vitessio#6737 - Update Vindex only on changes
vitessio#6765 - sort tabledefinitions before comparing them
vitessio#6781 - fixed orchestrator govet errors
vitessio#6772 - VStreamer: handle SET statements in the binlog stream
vitessio#6749 - Zero autoinc mode
vitessio#6779 - add myself to CODEOWNERS
vitessio#6774 - pitr: fix deadlock in ff code
vitessio#6776 - 7.0.2 Release Notes
vitessio#6733 - Materialize: Support rollups using a literal value as a column
vitessio#6744 - Refactor: Cleanup Noisy Workflow Logging
vitessio#6766 - Replaced Error with a warning in case parsing of VT_SPAN_CONTEXT fails
vitessio#6758 - operator precedence must take associativity into consideration
vitessio#6732 - upstream s3 ListBackups bug fix for vtbackup_cleaner
vitessio#6724 - Add if not exists support to database ddl in parsing
vitessio#6721 - Fixes long wait filter keyspace
vitessio#6714 - orc: more changes
vitessio#6728 - Remove auto-generation comment
vitessio#6730 - VReplication e2e tests: Simplify and split tests to overcome current CI resource constraints
vitessio#6726 - Reparent tests refactoring: setup/teardown for each test, reduce cut/paste, improve readability
vitessio#6741 - Revert "Make use of environment variables easier to troubleshoot"
vitessio#6708 - Evaluate system variables
vitessio#6735 - Make use of environment variables easier to troubleshoot
vitessio#6736 - Allow Update of same vindex using same vindex lookup on consistent lookup vindex
vitessio#6720 - Change tracker to use allprivs instead of dba
vitessio#6682 - Mysql -ssl tag is deprecated in the later versions of mysql
vitessio#6715 - e2e: deflake sharding tests
vitessio#6723 - helm & docker/k8s update
vitessio#6683 - Lock shard session heartbeat
vitessio#6700 - delete row using consistent lookup vindex in where clause
vitessio#6718 - make sure dual queries set found_rows correctly
vitessio#6680 - Support sql_calc_found_rows with limit in sharded keyspace
vitessio#6717 - maintainers: add shlomi-noach
vitessio#6525 - Field event: add additional attributes
vitessio#6494 - Allow set to have scope for each setting
vitessio#6495 - HandleSet in executor to handle multiple settings in single set statement
vitessio#6712 - Turn off schema tracker by default
vitessio#6710 - Fix unsaved event regression
vitessio#6695 - restore: checkNoDB should not require tables to be present
vitessio#6704 - docker/lite: Fix package URLs for ubi images.
vitessio#6672 - Add unit test case to improve test coverage for go/sqltypes/named_result.go
vitessio#6694 - Workflow test: fix flaky test. remove obsolete code
vitessio#6692 - Use statement support in olap mode
vitessio#6690 - add andres and harshit as code owners to query routing part
vitessio#6670 - VStream: mysql protocol
vitessio#6679 - restore: do not change tablet type to RESTORE if not actually performing a restore
vitessio#6673 - Reset Session for Reserved Connection Query Failure
vitessio#6674 - Try to install apt keys from list of different keyservers
vitessio#6667 - fix: information schema for non existent schema
vitessio#6666 - only take on simple dual queries in the vtgate
vitessio#6570 - Check http response code in vtctld API tests
vitessio#6613 - orc: vitess mode
vitessio#6624 - Clarify/expand MoveTables CLI help.
vitessio#6632 - Address vitessio#6631 by reducing optimization level
vitessio#6636 - build: remove chromium hard-coding
vitessio#6657 - Fix workflow CopyState fetching (and thus printing).
vitessio#6656 - fix path for vttablet query log in end to end test
vitessio#6568 - VRepl Collations: find collations for char primary keys and cast comparison during catchup
vitessio#6640 - VDiff: fix panic for tables with a unicode_loose_md5 vindex
vitessio#6653 - fix: allow passed table_schema in information_schema queries to vttablet if the keyspace is not found
vitessio#6649 - Add unit test for func Proto3ValuesEqual.
vitessio#6642 - AST struct name rewording
vitessio#6523 - Retry should not happen when in dedicated connection
vitessio#6648 - tm init: publish displayState
vitessio#6639 - tablet picker: use partial result from GetTabletMap if it is available
vitessio#6590 - Update some option docstrings for accuracy.
vitessio#6638 - Rewrite information_schema queries to use mysql db name instead of keyspace name
vitessio#6637 - Removed sqlparser.preview to set logstats stmtType and use plan.type
vitessio#6421 - VReplication: _vt.vreplication source column VARBINARY->BLOB
vitessio#6629 - Updates enforcement policy to match the one from the client
vitessio#6627 - Minor addition to Materialize help text.
vitessio#6594 - Implemented pitr testcase
vitessio#6615 - Rewrite SHOW TABLES
vitessio#6618 - tm: add more logging to checkMastership
vitessio#6544 - Workflow: List -> Show, and Expand Metadata
vitessio#6602 - 7.0.1 release notes
vitessio#6617 - Fix: Remove SetMaster Query Expectation from ERS Test
vitessio#6608 - tablet gateway: unit tests
vitessio#6610 - Add Sleep to ERS Unit Test
vitessio#6611 - Reverting package-lock.json to the one pre-PR vitessio#6603
vitessio#6582 - decider: initial import of Orchestrator
vitessio#6549 - Add unicode_loose_xxhash Vindex type
vitessio#6601 - Materialize: Only get schema from source tablets if target is missing tables
vitessio#6606 - Fix MoveTables docstring; was not valid JSON.
vitessio#6603 - vtctld UI: Fix logic for displaying vindexes
vitessio#6491 - GetSchema: Batch/parallel access to underlying mysqld for lower latency
vitessio#6487 - SET planning
vitessio#6586 - ForwardPort: Passthrough System Variables enabled on flag
vitessio#6596 - Remove long-unused memcache and cacheservice
vitessio#6598 - Fix vitessio#6597 by adding mysqld port display back to vtctld web UI
vitessio#6592 - Trivial copy-pasta comment fixup
vitessio#6565 - region_sharding: working resharding example
vitessio#6531 - vtgate routing cleanup: remove routeOptions
vitessio#6584 - vtcompose/docker-compose: fix InitShardMaster
vitessio#6552 - Close Idle reserved connections and Rollback Idle transactions
vitessio#6569 - Add realtime health stats to vtctld /api/keyspace/ks/tablets API
vitessio#6551 - Correctly report AUTOCOMMIT status in network packets
vitessio#6571 - build: replace gogo proto with golang
vitessio#6576 - Vtctl Workflow cmd: don't expect all shards to have workflow streams
vitessio#6580 - improve make proto
vitessio#6573 - Forward Port 6554 to master from 7.0
vitessio#6556 - Reverse workflow: update cells/tablet_types
vitessio#6579 - Make sure to handle EXPLAIN on tablets
vitessio#6546 - tablet_picker: keep trying to find a tablet until context expires
vitessio#6564 - tm: minor logging fix
vitessio#6522 - Remove shard session held in vtgate for reserved connection on connection failure
vitessio#6550 - Make sure to backtick schema names properly
vitessio#6517 - Lock Session Support
vitessio#6514 - Fix DDL Execution on reserved connection without in active transaction
vitessio#6536 - ensure tests for discoverygateway
vitessio#6535 - Add diagnostic logging to healthcheck.
vitessio#6526 - Minor vindex fixes
vitessio#6520 - replace more uses of slave with replica/replication
vitessio#6513 - 7.0.0 release notes
vitessio#6509 - Mini: fix tablet hostname
vitessio#6501 - handle boolean settings better
vitessio#6498 - Fixed typo
vitessio#6478 - [java] bump java version to 8.0.0-SNAPSHOT for next release
How this branch was constructed
First I determined where release-7.0 diverged from upstream's mainline:
Then I sanity checked it felt reasonable
And cut a branch for the release starting there (since new additions for 8.0 effectively begin from that point)
This is a bit of hand waving this a little because I forgot I wanted to land on this branch so I had to juggle branch names after the fact... but in practice
Now actually construct the PR; base it on
upstream/release-8.0And fold in our custom patches
Note that I did not cherry-pick Guido's change since it has been merged upstream