slack-vitess-r14.0.5: remove legacy sharding, add MySQL 8.x backup …

…test + compressors (vitessio#119) (vitessio#124)

* Delete all legacy sharding related code (vitessio#10278)

* Delete all legacy sharding related code



* Move used until functions from initialsharding to cluster



* Remove vtctl commands



* Kill vtworker and SetKeyspaceServedFrom cmd



* WaitForDrain related stragglers



* Legacy local straggler workflow



* Get rid of SetKeyspaceShardingInfo & wait for drain stragglers



* Remove vtworker stragglers



* Update throttlerservice protobuf



* Rename test 24, add hashicorp vault test to it (now mysql_server_vault)



* Remove last mentiones of legacy sharding in vtctl



* remove binlog_use_v3_resharding_mode



* Address review comments



* Address review comments



* Correct vtgate help output



* go fmt



* Remove v2 resharding fields (vitessio#10409)

* cleanup: remove sharding_column_name and sharding_column_type



* cleanup: remove sharding_column_name and sharding_column_type



* cleanup: remove sharding_column_name and sharding_column_type



* cleanup: remove sharding_column_name and sharding_column_type



* generate vtadmin files



* cleanup: remove sharding_column_name and sharding_column_type from vtadmin



* Merge from main



* Fix bad merge conflict resolution



* Fix missing 'sharding' import



* Fix bad conflict resolution in go/test/endtoend/cellalias/cell_alias_test.go



* Backup/Restore: add support for external compressors and decompressors (vitessio#10558)

* change to support an external decompressor




* add external compressor support + builtin additional compressors




* wrap external compressor/decompressor




* go mod tidy + comments




* add copyright notices




* add support for builtin engine




* Adding test case for buckup compression



* Fixing unit test and run mod tidy



* Removing unwanted unit tests



* Increase timeout of backup tests



* fixing linter errors



* Change test logic to accomodate running selective tests



* removing lint warning



* fixing test failure



* Removing un-necessary test



* Fixing code review feeback



* Change builtinEngine to consider 'auto' decompressor



* fixing Upgrade/Downgrade test



* Fix type & add summary under release notes



* Fixing typos in summary



* Fixing flag name typos






* Add MySQL 8 Support to Backup Tests (vitessio#10691)

* Add support for MySQL 8.0 in backup tests



* Add 8.0 workflow



* whitespace



* Use vtctldclient SetKeyspaceDurabilityPolicy to manage semi-sync

This needed to be done after the shard was setup in order to satisfy
the semantic assumptions related to semi-sync in the tests.



* Remove extraneous changes



* We need lz4 for TestXtrabackupStreamWithlz4Compression



* Try using Percona Repo for MySQL 8 to align mysqld and xtrabackup versions



* Specify stream type everywhere



* Remove repeated server install



* Moar...



* Move vtctlbackup test to 8.0



* Rename vtbackup test and move to MySQL 8



* Split the xbstream tests so the workflow doesn't time out

Otherwise it was going over the 10min limit and getting
killed.



* Use MySQL 8 compat method for setting passwords



* Test increasing timeout at another level



* Don't use the init passwords file with 8.0

mysqlctl doesn't start...

This is likely due to the change in initialization behavior with
MySQL 8.0 as it goes though two phases and you can't simply
start up mysqld and pass it data, it has to initialize and restart
first.



* Fix incorrect password update statement for vt_repl user



* Bump timeouts for 8.0 backup tests



* Bump it more :(



* Increase backup/restore timeout in backup_utils



* Apply new 8.0 template everywhere



* Fix bugs around how the compression flags were getting passed



* Use 45m timeout for the workflow, 30m for the run.



* These changes were no longer needed so limiting diff.



* Explicitly skip new linter check





* Fix test file merge issues



* Go 1.18.7 to fix error



* Fix lint



* Fix vttablet.txt



* Remove sharding_column_name and sharding_column_type from vtctld2 (vitessio#10459)

* Remove sharding_column_name and sharding_column_type



* Run make web_build to compile production files



* Remove vars from test



* Remove sharding column type select box test from web test



---------

Signed-off-by: Matt Lord <mattalord@gmail.com>
Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
Signed-off-by: Arvind Murty <arvind.murty@gmail.com>
Signed-off-by: notfelineit <notfelineit@gmail.com>
Co-authored-by: Matt Lord <mattalord@gmail.com>
Co-authored-by: Arvind Murty <arvind.murty@gmail.com>
Co-authored-by: Rameez Sajwani <rameezwazirali@hotmail.com>
Co-authored-by: Renan Rangel <rrangel@slack-corp.com>
Co-authored-by: Renan Rangel <renan@slack-corp.com>
Co-authored-by: Rohit Nayak <rohit@planetscale.com>
Co-authored-by: Frances Thai <31225471+notfelineit@users.noreply.github.com>

.github/workflows/cluster_endtoend_mysql80.yml

@@ -6,29 +6,23 @@ concurrency:
  group: format('{0}-{1}', ${{ github.ref }}, 'Cluster (mysql80)')
  cancel-in-progress: true
 
+env:
+ LAUNCHABLE_ORGANIZATION: "vitess"
+ LAUNCHABLE_WORKSPACE: "vitess-app"
+ GITHUB_PR_HEAD_SHA: "${{ github.event.pull_request.head.sha }}"
+
 jobs:
  build:
  name: Run endtoend tests on Cluster (mysql80)
  runs-on:
  group: vitess-ubuntu20
+ timeout-minutes: 45
 
  steps:
- - name: Check if workflow needs to be skipped
- id: skip-workflow
- run: |
- skip='false'
- if [[ "${{github.event.pull_request}}" == "" ]] && [[ "${{github.ref}}" != "refs/heads/main" ]] && [[ ! "${{github.ref}}" =~ ^refs/heads/slack-vitess-r[0-9]+\.[0-9]\.[0-9]+$ ]] && [[ ! "${{github.ref}}" =~ "refs/tags/.*" ]]; then
- skip='true'
- fi
- echo Skip ${skip}
- echo "skip-workflow=${skip}" >> $GITHUB_OUTPUT
-
  - name: Check out code
- if: steps.skip-workflow.outputs.skip-workflow == 'false'
- uses: actions/checkout@v3
+ uses: actions/checkout@v2
 
  - name: Check for changes in relevant files
- if: steps.skip-workflow.outputs.skip-workflow == 'false'
  uses: frouioui/paths-filter@main
  id: changes
  with:
@@ -39,74 +33,84 @@ jobs:
  - 'test.go'
  - 'Makefile'
  - 'build.env'
- - 'go.sum'
- - 'go.mod'
+ - 'go.[sumod]'
  - 'proto/*.proto'
  - 'tools/**'
  - 'config/**'
  - 'bootstrap.sh'
- - '.github/workflows/cluster_endtoend_mysql80.yml'
+ - '.github/workflows/**'
 
  - name: Set up Go
- if: steps.skip-workflow.outputs.skip-workflow == 'false' && steps.changes.outputs.end_to_end == 'true'
- uses: actions/setup-go@v3
+ if: steps.changes.outputs.end_to_end == 'true'
+ uses: actions/setup-go@v2
  with:
  go-version: 1.19.10
 
  - name: Set up python
- if: steps.skip-workflow.outputs.skip-workflow == 'false' && steps.changes.outputs.end_to_end == 'true'
- uses: actions/setup-python@v4
+ if: steps.changes.outputs.end_to_end == 'true'
+ uses: actions/setup-python@v2
 
  - name: Tune the OS
- if: steps.skip-workflow.outputs.skip-workflow == 'false' && steps.changes.outputs.end_to_end == 'true'
+ if: steps.changes.outputs.end_to_end == 'true'
  run: |
- # Limit local port range to not use ports that overlap with server side
- # ports that we listen on.
  echo '1024 65535' | sudo tee -a /proc/sys/net/ipv4/ip_local_port_range
  # Increase the asynchronous non-blocking I/O. More information at https://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html#sysvar_innodb_use_native_aio
  echo "fs.aio-max-nr = 1048576" | sudo tee -a /etc/sysctl.conf
  sudo sysctl -p /etc/sysctl.conf
 
  - name: Get dependencies
- if: steps.skip-workflow.outputs.skip-workflow == 'false' && steps.changes.outputs.end_to_end == 'true'
+ if: steps.changes.outputs.end_to_end == 'true'
  run: |
- 
- # Get key to latest MySQL repo
- sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 467B942D3A79BD29
- # Setup MySQL 8.0
- wget -c https://dev.mysql.com/get/mysql-apt-config_0.8.20-1_all.deb
- echo mysql-apt-config mysql-apt-config/select-server select mysql-8.0 | sudo debconf-set-selections
- sudo DEBIAN_FRONTEND="noninteractive" dpkg -i mysql-apt-config*
+ # Setup Percona Server for MySQL 8.0
  sudo apt-get update
- # Install everything else we need, and configure
- sudo apt-get install -y mysql-server mysql-client make unzip g++ etcd curl git wget eatmydata xz-utils
+ sudo apt-get install -y lsb-release gnupg2 curl
+ wget https://repo.percona.com/apt/percona-release_latest.$(lsb_release -sc)_all.deb
+ sudo DEBIAN_FRONTEND="noninteractive" dpkg -i percona-release_latest.$(lsb_release -sc)_all.deb
+ sudo percona-release setup ps80
  sudo apt-get update
- 
- sudo service etcd stop
- echo 'mysql version: '
- sudo mysql --version
+
+ # Install everything else we need, and configure
+ sudo apt-get install -y percona-server-server percona-server-client make unzip g++ etcd git wget eatmydata xz-utils
  sudo service mysql stop
+ sudo service etcd stop
  sudo ln -s /etc/apparmor.d/usr.sbin.mysqld /etc/apparmor.d/disable/
  sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.mysqld
  go mod download
 
  # install JUnit report formatter
  go install github.com/vitessio/go-junit-report@HEAD
 
+ - name: Setup launchable dependencies
+ if: steps.changes.outputs.end_to_end == 'true'
+ run: |
+ # Get Launchable CLI installed. If you can, make it a part of the builder image to speed things up
+ pip3 install --user launchable~=1.0 > /dev/null
+
+ # verify that launchable setup is all correct.
+ launchable verify || true
+
+ # Tell Launchable about the build you are producing and testing
+ launchable record build --name "$GITHUB_RUN_ID" --source .
+
  - name: Run cluster endtoend test
- if: steps.skip-workflow.outputs.skip-workflow == 'false' && steps.changes.outputs.end_to_end == 'true'
- uses: nick-fields/retry@v2
- with:
- timeout_minutes: 45
- max_attempts: 3
- retry_on: error
- command: |
- # We set the VTDATAROOT to the /tmp folder to reduce the file path of mysql.sock file
- # which musn't be more than 107 characters long.
- export VTDATAROOT="/tmp/"
- source build.env
-
- set -x
-
- # run the tests however you normally do, then produce a JUnit XML file
- eatmydata -- go run test.go -docker=false -follow -shard mysql80
+ if: steps.changes.outputs.end_to_end == 'true'
+ timeout-minutes: 30
+ run: |
+ # We set the VTDATAROOT to the /tmp folder to reduce the file path of mysql.sock file
+ # which musn't be more than 107 characters long.
+ export VTDATAROOT="/tmp/"
+ source build.env
+
+ set -x
+
+ # run the tests however you normally do, then produce a JUnit XML file
+ eatmydata -- go run test.go -docker=false -follow -shard mysql80 | tee -a output.txt | go-junit-report -set-exit-code > report.xml
+
+ - name: Print test output and Record test result in launchable
+ if: steps.changes.outputs.end_to_end == 'true' && always()
+ run: |
+ # send recorded tests to launchable
+ launchable record tests --build "$GITHUB_RUN_ID" go-test . || true
+
+ # print test output
+ cat output.txt

.github/workflows/cluster_endtoend_24.yml → .github/workflows/cluster_endtoend_mysql_server_vault.yml

@@ -1,14 +1,14 @@
 # DO NOT MODIFY: THIS FILE IS GENERATED USING "make generate_ci_workflows"
 
-name: Cluster (24)
+name: Cluster (mysql_server_vault)
 on: [push, pull_request]
 concurrency:
- group: format('{0}-{1}', ${{ github.ref }}, 'Cluster (24)')
+ group: format('{0}-{1}', ${{ github.ref }}, 'Cluster (mysql_server_vault)')
  cancel-in-progress: true
 
 jobs:
  build:
- name: Run endtoend tests on Cluster (24)
+ name: Run endtoend tests on Cluster (mysql_server_vault)
  runs-on:
  group: vitess-ubuntu20
 
@@ -113,5 +113,14 @@ jobs:
 
  set -x
 
- # run the tests however you normally do, then produce a JUnit XML file
- eatmydata -- go run test.go -docker=false -follow -shard 24
+ # run the tests however you normally do, then produce a JUnit XML file
+ eatmydata -- go run test.go -docker=false -follow -shard mysql_server_vault | tee -a output.txt | go-junit-report -set-exit-code > report.xml
+
+ - name: Print test output and Record test result in launchable
+ if: steps.changes.outputs.end_to_end == 'true' && always()
+ run: |
+ # send recorded tests to launchable
+ launchable record tests --build "$GITHUB_RUN_ID" go-test . || true
+
+ # print test output
+ cat output.txt