xtrabackup implementation with testcases #4695

deepthi · 2019-03-02T01:46:50Z

Feature Description

This PR adds support for xtrabackup backups through Vitess.
Backups and restores can be executed just as they are today, the vtctl commands don't change.

Pre-requisites, Supported Versions

A compatible version of xtrabackup (and xbstream if using) must be present in PATH. For MySQL 5.7 and MariaDB 10, the xtrabackup versions that are allowed are 2.4.x and lower. For MySQL 8.0, it is 8.0.x.
MariaDB 10.3 is not compatible with xtrabackup.

How to Use

The change is to vttablet startup. vttablet needs to be told to use xtrabackup instead of the current method (which we are calling builtin), and provided the necessary flags. The vttablet flag is
-backup_engine_implementation=xtrabackup

Features

Only streaming mode backups are supported right now.
All 4 currently supported backup storage types will be supported by xtrabackup streaming backups also.
A MANIFEST file is created along with the backup archive, which can be either tar or xbstream.
gzip compression and uncompression is supported for xtrabackup just like for the current backups.

Options

Additional vttablet flags:

-xtrabackup_user: Required. database user to be used for backup
-xtrabackup_backup_flags: Optional. flags to add to the xtrabackup backup command
-xtrabackup_stream_mode: Optional. Valid values are tar and xbstream. Defaults to tar.
-xbstream_flags: Optional. Passed to xbstream command during restore.
If additional flags are passed to xtrabackup and xbstream, care must be taken that they match (compress/uncompress, encrypt/decrypt etc.)

Tasks

Before we are ready to merge this, we need to
~~move the backup_engine_implementation choice from vttablet to vtctl~~
See #3957 (comment)

use vterrors to wrap errors
~~implement a backup user dbConfig~~
add -xtrabackup_user command line parameter
add xtrabackup to docker containers
get tests passing in docker
functional testing

Fixes #3957
Signed-off-by: deepthi deepthi@planetscale.com

config/mycnf/default-fast.cnf

go/vt/dbconfigs/dbconfigs.go

go/vt/vttablet/tabletmanager/rpc_backup.go

go/vt/mysqlctl/xtrabackupengine.go

rafael · 2019-04-04T17:04:12Z

@deepthi - This is looking great. Using the xtrabackup manifest simplified the logic in the engine.

I added couple nit comments on some of the flags descriptions.

go/vt/mysqlctl/xtrabackupengine.go

docker/bootstrap/Dockerfile.mysql56

docker/bootstrap/Dockerfile.mariadb

docker/bootstrap/Dockerfile.mysql80

go/vt/mysqlctl/backupengine.go

go/vt/mysqlctl/xtrabackupengine.go

Signed-off-by: deepthi <deepthi@planetscale.com>

…k warnings Signed-off-by: deepthi <deepthi@planetscale.com>

…storing xtrabackup Signed-off-by: deepthi <deepthi@planetscale.com>

Signed-off-by: deepthi <deepthi@planetscale.com>

… using xtrabackup, loop through existing backups to find the most recent good backup Signed-off-by: deepthi <deepthi@planetscale.com>

…ckup and restore Signed-off-by: deepthi <deepthi@planetscale.com>

Signed-off-by: deepthi <deepthi@planetscale.com>

…r review comments Signed-off-by: deepthi <deepthi@planetscale.com>

teejae · 2019-04-09T02:05:02Z

test/backup.py

+ if use_xtrabackup:
+ xtra_args.extend(xtrabackup_args)
+
+ hook_args = ['-backup_storage_hook',


Suggested change

hook_args = ['-backup_storage_hook',

hook_args = ['-backup_storage_hook=test_backup_transform',

'-backup_storage_compress=false']

rafael

LGTM +1

One thing that caught my attention from the last commit is this function: findReplicationPosition. I think we should consider testing that in more depth. It seems to me that this part is still fragile. I wouldn't block for this right now.

deepthi · 2019-04-11T17:27:33Z

LGTM +1

One thing that caught my attention from the last commit is this function: findReplicationPosition. I think we should consider testing that in more depth. It seems to me that this part is still fragile. I wouldn't block for this right now.

Good point. We have an integration test written specifically for this part, but we should also test some failure cases.

@demmer

It's been busy and exciting in vitess upstream. Some changes that felt worth calling out as they could impact assumptions about behavior: - vitessio#4832 — this adds a timeout that is 3x the `healthCheckInterval` which at which point the tablet will stop reporting as healthy - vitessio#4814 — this changes when command line args are used by vttablet - vitessio#4811 — a planned reparent now sets `super_read_only` - vitessio#4805 — behavior coming back from a failed backup is different **Slack changes** @demmer vitessio#4827 Correct suppress logging for begin...commit in autocommit @rafael vitessio#4824 adds timeouts for all statements @demmer vitessio#4826 adds logging of stack traces as opt-in @demmer vitessio#4819 remove begin/commit logs from autocommit txns @demmer vitessio#4796 improves support for vtexplain for begin/dml/dml/commit txns **Non slack changes** vitessio#4839 Improve behavior with reference table routing & vreplication vitessio#4833 Support query routing given there could be multiple targets for a table vitessio#4832 tablets get new health check behavior (health checks time out) vitessio#4837 refresh bug around row streamer handling creds vitessio#4830 apply a default value to the db_name attribute in local_metadata vitessio#4785 SHOW SCHEMAS aliased to SHOW DATABASES vitessio#4829 add experimental support for split clone & vertical split clone via vreplication vitessio#4822 do not normalize in order by vitessio#4791 Adds vreplication row streaming support vitessio#4814 command line flags now used regardless of management mode vitessio#4811 vttablet sets super_read_only during planned reparent vitessio#4803 xtrabackup testing vitessio#4727 support multiple vttablets running against one mysql instance vitessio#4746 TopoCat can now produce JSON output vitessio#4805 mysqld is now restarted after a failed backup vitessio#4685 macos bootstrap bug vitessio#4874 ZK opts and java land vitessio#4695 Adds support for xtrabackup vitessio#4794 fixes build failures vitessio#4725 changes mysql8.0 start args vitessio#4736 introduce new states to support vreplication vitessio#4788 expands orc error **Docs etc** vitessio#4831 lint vitessio#4827 docs vitessio#4816 const declaration cleanup vitessio#4820 const declaration cleanup vitessio#4825 docs vitessio#4818 docs vitessio#4809 docs vitessio#4812 moves consts around vitessio#4813 docs vitessio#4808 docs vitessio#4800 docs vitessio#4795 docs

deepthi mentioned this pull request Mar 2, 2019

WIP: xtrabackup implementation + allow_master #4614

Closed

deepthi force-pushed the ds-xtrabackup branch 3 times, most recently from 072eb8f to b02ed04 Compare March 7, 2019 22:20

rafael reviewed Mar 12, 2019

View reviewed changes

deepthi force-pushed the ds-xtrabackup branch from f2f9dc6 to 80bdefb Compare March 13, 2019 20:12

deepthi force-pushed the ds-xtrabackup branch from b8b27e3 to 9e0f0a8 Compare March 21, 2019 01:47

deepthi marked this pull request as ready for review March 25, 2019 17:01

deepthi requested review from derekperkins, dkhenry and sougou as code owners March 25, 2019 17:01

rafael reviewed Mar 25, 2019

View reviewed changes

go/vt/mysqlctl/xtrabackupengine.go Outdated Show resolved Hide resolved

rafael reviewed Mar 26, 2019

View reviewed changes

go/vt/mysqlctl/xtrabackupengine.go Outdated Show resolved Hide resolved

deepthi force-pushed the ds-xtrabackup branch from 97f59cc to 69635d2 Compare March 31, 2019 00:07

rafael reviewed Apr 4, 2019

View reviewed changes

go/vt/mysqlctl/xtrabackupengine.go Outdated Show resolved Hide resolved

go/vt/mysqlctl/xtrabackupengine.go Outdated Show resolved Hide resolved

teejae suggested changes Apr 8, 2019

View reviewed changes

deepthi added 13 commits April 8, 2019 10:17

xtrabackup implementation with testcases

a434720

Signed-off-by: deepthi <deepthi@planetscale.com>

implement vttablet staying in SERVING state during backup

9c3cd81

Signed-off-by: deepthi <deepthi@planetscale.com>

create MANIFEST, use vterrors

9a647b9

Signed-off-by: deepthi <deepthi@planetscale.com>

add xtrabackup to docker containers

c1fc395

Signed-off-by: deepthi <deepthi@planetscale.com>

add position and cmd line params to manifest

3d8442a

Signed-off-by: deepthi <deepthi@planetscale.com>

fix failing test by putting it in a separate suite

c3c48be

Signed-off-by: deepthi <deepthi@planetscale.com>

fix scoping bug that causes backup to hang

706807f

Signed-off-by: deepthi <deepthi@planetscale.com>

add xtrabackup_user command line option, use vterrors, fix staticchec…

b8112ed

…k warnings Signed-off-by: deepthi <deepthi@planetscale.com>

make MySQL56 default flavor for parsing replication position while re…

3b96754

…storing xtrabackup Signed-off-by: deepthi <deepthi@planetscale.com>

pass --target-dir to xtrabackup

359fbe2

Signed-off-by: deepthi <deepthi@planetscale.com>

handle gtid correctly during restore, add logging for testing

d6e446f

Signed-off-by: deepthi <deepthi@planetscale.com>

better option documentation, make xtrabackupUser a required param for…

39c859b

… using xtrabackup, loop through existing backups to find the most recent good backup Signed-off-by: deepthi <deepthi@planetscale.com>

testcase to verify that GTIDSet is handled correctly by xtrabackup ba…

d6dc854

…ckup and restore Signed-off-by: deepthi <deepthi@planetscale.com>

deepthi added 2 commits April 8, 2019 10:17

detect MySQL_FLAVOR from a db connection instead of env variable

e918b5c

Signed-off-by: deepthi <deepthi@planetscale.com>

use xtrabackup exit status to determine success/failure, address othe…

1cd2ad1

…r review comments Signed-off-by: deepthi <deepthi@planetscale.com>

deepthi force-pushed the ds-xtrabackup branch from 69635d2 to 1cd2ad1 Compare April 8, 2019 23:48

teejae suggested changes Apr 9, 2019

View reviewed changes

rafael approved these changes Apr 10, 2019

View reviewed changes

deepthi merged commit f3bfe8b into vitessio:master Apr 11, 2019

setassociative mentioned this pull request Apr 26, 2019

Merge upstream 2019-04-25 tinyspeck/vitess#130

Merged

deepthi mentioned this pull request Oct 11, 2019

xtrabackup is not documented vitessio/website#261

Closed

harshit-gangal deleted the ds-xtrabackup branch July 17, 2020 16:39

systay pushed a commit that referenced this pull request Jul 22, 2024

cherry pick of 14871 (#4695)

8573ec0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

xtrabackup implementation with testcases #4695

xtrabackup implementation with testcases #4695

deepthi commented Mar 2, 2019 •

edited

Loading

rafael commented Apr 4, 2019

teejae Apr 9, 2019

rafael left a comment

deepthi commented Apr 11, 2019

	hook_args = ['-backup_storage_hook',
	hook_args = ['-backup_storage_hook=test_backup_transform',
	'-backup_storage_compress=false']

xtrabackup implementation with testcases #4695

xtrabackup implementation with testcases #4695

Conversation

deepthi commented Mar 2, 2019 • edited Loading

Feature Description

Pre-requisites, Supported Versions

How to Use

Features

Options

Tasks

rafael commented Apr 4, 2019

teejae Apr 9, 2019

Choose a reason for hiding this comment

rafael left a comment

Choose a reason for hiding this comment

deepthi commented Apr 11, 2019

deepthi commented Mar 2, 2019 •

edited

Loading