Skip to content

Commit

Permalink
[latest-17.0](vitessio#2526): CherryPick(vitessio#13238): VReplicatio…
Browse files Browse the repository at this point in the history
…n Workflows: make sequence tables follow routing rules (vitessio#2528)

* backport of 2526

* Adjust new workflows for vitess-private

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Use basic vs simple everywhere for new workflow

Signed-off-by: Matt Lord <mattalord@gmail.com>

---------

Signed-off-by: Matt Lord <mattalord@gmail.com>
Co-authored-by: Matt Lord <mattalord@gmail.com>
  • Loading branch information
planetscale-actions-bot and mattlord authored Jul 4, 2023
1 parent 474e942 commit 0c60f38
Show file tree
Hide file tree
Showing 36 changed files with 2,593 additions and 469 deletions.
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
# DO NOT MODIFY: THIS FILE IS GENERATED USING "make generate_ci_workflows"

name: Cluster (vreplication_partial_movetables_basic)
on: [push]
concurrency:
group: format('{0}-{1}', ${{ github.ref }}, 'Cluster (vreplication_partial_movetables_basic)')
cancel-in-progress: true

permissions: read-all

jobs:
build:
name: Run endtoend tests on Cluster (vreplication_partial_movetables_basic)
runs-on: ubuntu-22.04

steps:
- name: Configure git private repo access
env:
GITHUB_TOKEN: ${{ secrets.PLANETSCALE_ACTIONS_BOT_TOKEN }}
run: |
git config --global --add url."https://${GITHUB_TOKEN}@github.com/".insteadOf "https://github.com/"
- name: Check out code
uses: actions/checkout@v3

- name: Check for changes in relevant files
uses: frouioui/paths-filter@main
id: changes
with:
token: ''
filters: |
end_to_end:
- 'go/**/*.go'
- 'test.go'
- 'Makefile'
- 'build.env'
- 'go.sum'
- 'go.mod'
- 'proto/*.proto'
- 'tools/**'
- 'config/**'
- 'bootstrap.sh'
- '.github/workflows/cluster_endtoend_vreplication_partial_movetables_basic.yml'
- name: Set up Go
uses: actions/setup-go@v4
with:
go-version: 1.20.3

- name: Set up python
uses: actions/setup-python@v4

- name: Tune the OS
run: |
# Limit local port range to not use ports that overlap with server side
# ports that we listen on.
sudo sysctl -w net.ipv4.ip_local_port_range="22768 65535"
# Increase the asynchronous non-blocking I/O. More information at https://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html#sysvar_innodb_use_native_aio
echo "fs.aio-max-nr = 1048576" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p /etc/sysctl.conf
- name: Get dependencies
env: # Or as an environment variable
AWS_ACCESS_KEY_ID: ${{ secrets.BUILDKITE_S3_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.BUILDKITE_S3_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: us-east-1
run: |
# Get key to latest MySQL repo
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 467B942D3A79BD29
# Setup MySQL 8.0
wget -c https://dev.mysql.com/get/mysql-apt-config_0.8.24-1_all.deb
echo mysql-apt-config mysql-apt-config/select-server select mysql-8.0 | sudo debconf-set-selections
sudo DEBIAN_FRONTEND="noninteractive" dpkg -i mysql-apt-config*
sudo apt-get update
# Install everything else we need, and configure
sudo apt-get install -y mysql-server mysql-client make unzip g++ etcd curl git wget eatmydata xz-utils libncurses5
sudo service mysql stop
sudo service etcd stop
sudo ln -s /etc/apparmor.d/usr.sbin.mysqld /etc/apparmor.d/disable/
sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.mysqld
go mod download
- name: Run cluster endtoend test
timeout-minutes: 45
run: |
# We set the VTDATAROOT to the /tmp folder to reduce the file path of mysql.sock file
# which musn't be more than 107 characters long.
export VTDATAROOT="/tmp/"
source build.env
set -x
# Increase our open file descriptor limit as we could hit this
ulimit -n 65536
cat <<-EOF>>./config/mycnf/mysql80.cnf
innodb_buffer_pool_dump_at_shutdown=OFF
innodb_buffer_pool_in_core_file=OFF
innodb_buffer_pool_load_at_startup=OFF
innodb_buffer_pool_size=64M
innodb_doublewrite=OFF
innodb_flush_log_at_trx_commit=0
innodb_flush_method=O_DIRECT
innodb_numa_interleave=ON
innodb_adaptive_hash_index=OFF
sync_binlog=0
sync_relay_log=0
performance_schema=OFF
slow-query-log=OFF
EOF
cat <<-EOF>>./config/mycnf/mysql80.cnf
binlog-transaction-compression=ON
EOF
# run the tests however you normally do
eatmydata -- go run test.go -docker=false -follow -shard vreplication_partial_movetables_basic
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
# DO NOT MODIFY: THIS FILE IS GENERATED USING "make generate_ci_workflows"

name: Cluster (vreplication_partial_movetables_sequences)
on: [push]
concurrency:
group: format('{0}-{1}', ${{ github.ref }}, 'Cluster (vreplication_partial_movetables_sequences)')
cancel-in-progress: true

permissions: read-all

jobs:
build:
name: Run endtoend tests on Cluster (vreplication_partial_movetables_sequences)
runs-on: ubuntu-22.04

steps:
- name: Configure git private repo access
env:
GITHUB_TOKEN: ${{ secrets.PLANETSCALE_ACTIONS_BOT_TOKEN }}
run: |
git config --global --add url."https://${GITHUB_TOKEN}@github.com/".insteadOf "https://github.com/"
- name: Check out code
uses: actions/checkout@v3

- name: Check for changes in relevant files
uses: frouioui/paths-filter@main
id: changes
with:
token: ''
filters: |
end_to_end:
- 'go/**/*.go'
- 'test.go'
- 'Makefile'
- 'build.env'
- 'go.sum'
- 'go.mod'
- 'proto/*.proto'
- 'tools/**'
- 'config/**'
- 'bootstrap.sh'
- '.github/workflows/cluster_endtoend_vreplication_partial_movetables_sequences.yml'
- name: Set up Go
uses: actions/setup-go@v4
with:
go-version: 1.20.3

- name: Set up python
uses: actions/setup-python@v4

- name: Tune the OS
run: |
# Limit local port range to not use ports that overlap with server side
# ports that we listen on.
sudo sysctl -w net.ipv4.ip_local_port_range="22768 65535"
# Increase the asynchronous non-blocking I/O. More information at https://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html#sysvar_innodb_use_native_aio
echo "fs.aio-max-nr = 1048576" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p /etc/sysctl.conf
- name: Get dependencies
env: # Or as an environment variable
AWS_ACCESS_KEY_ID: ${{ secrets.BUILDKITE_S3_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.BUILDKITE_S3_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: us-east-1
run: |
# Get key to latest MySQL repo
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 467B942D3A79BD29
# Setup MySQL 8.0
wget -c https://dev.mysql.com/get/mysql-apt-config_0.8.24-1_all.deb
echo mysql-apt-config mysql-apt-config/select-server select mysql-8.0 | sudo debconf-set-selections
sudo DEBIAN_FRONTEND="noninteractive" dpkg -i mysql-apt-config*
sudo apt-get update
# Install everything else we need, and configure
sudo apt-get install -y mysql-server mysql-client make unzip g++ etcd curl git wget eatmydata xz-utils libncurses5
sudo service mysql stop
sudo service etcd stop
sudo ln -s /etc/apparmor.d/usr.sbin.mysqld /etc/apparmor.d/disable/
sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.mysqld
go mod download
- name: Run cluster endtoend test
timeout-minutes: 45
run: |
# We set the VTDATAROOT to the /tmp folder to reduce the file path of mysql.sock file
# which musn't be more than 107 characters long.
export VTDATAROOT="/tmp/"
source build.env
set -x
# Increase our open file descriptor limit as we could hit this
ulimit -n 65536
cat <<-EOF>>./config/mycnf/mysql80.cnf
innodb_buffer_pool_dump_at_shutdown=OFF
innodb_buffer_pool_in_core_file=OFF
innodb_buffer_pool_load_at_startup=OFF
innodb_buffer_pool_size=64M
innodb_doublewrite=OFF
innodb_flush_log_at_trx_commit=0
innodb_flush_method=O_DIRECT
innodb_numa_interleave=ON
innodb_adaptive_hash_index=OFF
sync_binlog=0
sync_relay_log=0
performance_schema=OFF
slow-query-log=OFF
EOF
cat <<-EOF>>./config/mycnf/mysql80.cnf
binlog-transaction-compression=ON
EOF
# run the tests however you normally do
eatmydata -- go run test.go -docker=false -follow -shard vreplication_partial_movetables_sequences
31 changes: 27 additions & 4 deletions changelog/18.0/18.0.0/summary.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@
- [Deleted `vtgr`](#deleted-vtgr)
- **[New stats](#new-stats)**
- [VTGate Vindex unknown parameters](#vtgate-vindex-unknown-parameters)
- **[VTTablet](#vttablet)**
- [VTTablet: New ResetSequences RPC](#vttablet-new-rpc-reset-sequences)

## <a id="major-changes"/>Major Changes

Expand All @@ -20,15 +22,25 @@

#### <a id="new-flag-toggle-ers"/>VTOrc flag `--allow-emergency-reparent`

VTOrc has a new flag `--allow-emergency-reparent` that allows the users to toggle the ability of VTOrc to run emergency reparent operations.
The users that want VTOrc to fix the replication issues, but don't want it to run any reparents should start using this flag.
By default, VTOrc will be able to run `EmergencyReparentShard`. The users must specify the flag to `false` to change the behaviour.
VTOrc has a new flag `--allow-emergency-reparent` that allows the users to toggle the ability of VTOrc to run emergency
reparent operations. The users that want VTOrc to fix the replication issues, but don't want it to run any reparents
should start using this flag. By default, VTOrc will be able to run `EmergencyReparentShard`. The users must specify the
flag to `false` to change the behaviour.

### <a id="vtadmin"/>VTAdmin

#### <a id="updated-node"/>vtadmin-web updated to node v18.16.0 (LTS)

Building vtadmin-web now requires node >= v18.16.0 (LTS). Breaking changes from v16 to v18 are listed
in https://nodejs.org/en/blog/release/v18.0.0, but none apply to VTAdmin. Full details on v18.16.0 are listed
here https://nodejs.org/en/blog/release/v18.16.0.

### <a id="deprecations-and-deletions"/>Deprecations and Deletions

#### <a id="deleted-k8stopo"/>Deleted `k8stopo`

The `k8stopo` has been deprecated in Vitess 17, also see https://github.com/vitessio/vitess/issues/13298. With Vitess 18 the `k8stopo` has been removed.
The `k8stopo` has been deprecated in Vitess 17, also see https://github.com/vitessio/vitess/issues/13298. With Vitess 18
the `k8stopo` has been removed.

#### <a id="deleted-vtgr"/>Deleted `vtgr`

Expand All @@ -39,3 +51,14 @@ The `vtgr` has been deprecated in Vitess 17, also see https://github.com/vitessi
#### <a id="vtgate-vindex-unknown-parameters"/>VTGate Vindex unknown parameters

The VTGate stat `VindexUnknownParameters` gauges unknown Vindex parameters found in the latest VSchema pulled from the topology.

### <a id="vttablet"/>VTTablet

#### <a id="vttablet-new-rpc-reset-sequences"/>New ResetSequences rpc

A new vttablet RPC `ResetSequences` has been added, which is being used by `MoveTables` and `Migrate` for workflows
where a `sequence` table is being moved (https://github.com/vitessio/vitess/pull/13238). This has an impact on the
Vitess upgrade process from an earlier version if you need to use such a workflow before the entire cluster is upgraded.

Any MoveTables or Migrate workflow that moves a sequence table should only be run after all vitess components have been
upgraded, and no upgrade should be done while such a workflow is in progress.
1 change: 1 addition & 0 deletions changelog/README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
## Releases
* [18.0](18.0)
* [17.0](17.0)
* [16.0](16.0)
* [15.0](15.0)
Expand Down
7 changes: 0 additions & 7 deletions go/test/endtoend/messaging/r

This file was deleted.

Loading

0 comments on commit 0c60f38

Please sign in to comment.