get TS and use it before full backup using mydumper #534

weekface · 2019-05-29T07:00:31Z

What problem does this PR solve?

This pr fixes: #533 and closes: #540

remove --chunk-filesize option
set --verbose to 3 (info)
fix stability tests about binlog backup
get TS before backup

The mydumper output when we change --verbose to 3:

** Message: Server version reported as: 5.7.25-TiDB-v3.0.0-rc.1
** Message: Connected to a TiDB server
** Message: Skipping locks because of TiDB
** Message: Set to tidb_snapshot '408711625696608257'
** Message: Started dump at: 2019-05-29 06:25:37
** Message: Written master status
** Message: Thread 1 connected using MySQL connection ID 28
** Message: Thread 1 set to tidb_snapshot '408711625696608257'
** Message: Thread 2 connected using MySQL connection ID 60
** Message: Thread 2 set to tidb_snapshot '408711625696608257'
** Message: Thread 3 connected using MySQL connection ID 29
** Message: Thread 3 set to tidb_snapshot '408711625696608257'
** Message: Thread 4 connected using MySQL connection ID 61
** Message: Thread 4 set to tidb_snapshot '408711625696608257'
** Message: Thread 3 dumping data for `mysql`.`GLOBAL_VARIABLES`
** Message: Non-InnoDB dump complete, unlocking tables
** Message: Thread 2 dumping data for `mysql`.`columns_priv`
** Message: Thread 1 dumping data for `mysql`.`bind_info`
** Message: Thread 4 dumping data for `mysql`.`db`
** Message: Empty table mysql.bind_info
** Message: Thread 1 dumping data for `mysql`.`default_roles`
** Message: Empty table mysql.columns_priv
** Message: Thread 2 dumping data for `mysql`.`gc_delete_range`
** Message: Empty table mysql.db
** Message: Thread 4 dumping data for `mysql`.`gc_delete_range_done`
** Message: Thread 3 dumping data for `mysql`.`help_topic`
** Message: Empty table mysql.gc_delete_range
** Message: Thread 2 dumping data for `mysql`.`role_edges`
** Message: Empty table mysql.gc_delete_range_done
** Message: Thread 4 dumping data for `mysql`.`stats_buckets`
** Message: Empty table mysql.help_topic
** Message: Thread 3 dumping data for `mysql`.`stats_feedback`
** Message: Empty table mysql.role_edges
** Message: Thread 2 dumping data for `mysql`.`stats_histograms`
** Message: Empty table mysql.stats_feedback
** Message: Thread 3 dumping data for `mysql`.`stats_meta`
** Message: Thread 3 dumping data for `mysql`.`stats_top_n`
** Message: Empty table mysql.stats_top_n
** Message: Thread 3 dumping data for `mysql`.`tables_priv`
** Message: Thread 2 dumping data for `mysql`.`tidb`
** Message: Thread 2 dumping data for `mysql`.`user`
** Message: Thread 2 dumping data for `test`.`block_writer`
** Message: Thread 4 dumping data for `test`.`block_writer1`
** Message: Empty table mysql.default_roles
** Message: Thread 1 dumping data for `test`.`block_writer10`
** Message: Thread 4 dumping data for `test`.`block_writer11`
** Message: Thread 2 dumping data for `test`.`block_writer12`
** Message: Thread 1 dumping data for `test`.`block_writer13`
** Message: Empty table mysql.tables_priv
** Message: Thread 3 dumping data for `test`.`block_writer14`
** Message: Thread 4 dumping data for `test`.`block_writer15`
** Message: Thread 2 dumping data for `test`.`block_writer16`
** Message: Thread 1 dumping data for `test`.`block_writer17`
** Message: Thread 3 dumping data for `test`.`block_writer18`
** Message: Thread 4 dumping data for `test`.`block_writer19`
** Message: Thread 4 dumping data for `test`.`block_writer2`
** Message: Thread 3 dumping data for `test`.`block_writer20`
** Message: Thread 2 dumping data for `test`.`block_writer21`
** Message: Thread 4 dumping data for `test`.`block_writer22`
** Message: Thread 2 dumping data for `test`.`block_writer23`
** Message: Thread 3 dumping data for `test`.`block_writer24`
** Message: Thread 1 dumping data for `test`.`block_writer25`
** Message: Thread 4 dumping data for `test`.`block_writer26`
** Message: Thread 3 dumping data for `test`.`block_writer27`
** Message: Thread 2 dumping data for `test`.`block_writer28`
** Message: Thread 1 dumping data for `test`.`block_writer29`
** Message: Thread 4 dumping data for `test`.`block_writer3`
** Message: Thread 3 dumping data for `test`.`block_writer30`
** Message: Thread 2 dumping data for `test`.`block_writer31`
** Message: Thread 1 dumping data for `test`.`block_writer32`
** Message: Thread 4 dumping data for `test`.`block_writer33`
** Message: Thread 2 dumping data for `test`.`block_writer34`
** Message: Thread 4 dumping data for `test`.`block_writer35`
** Message: Thread 3 dumping data for `test`.`block_writer36`
** Message: Thread 2 dumping data for `test`.`block_writer37`
** Message: Thread 4 dumping data for `test`.`block_writer38`
** Message: Thread 1 dumping data for `test`.`block_writer39`
** Message: Thread 3 dumping data for `test`.`block_writer4`
** Message: Thread 2 dumping data for `test`.`block_writer40`
** Message: Thread 4 dumping data for `test`.`block_writer41`
** Message: Thread 1 dumping data for `test`.`block_writer42`
** Message: Thread 3 dumping data for `test`.`block_writer43`
** Message: Thread 2 dumping data for `test`.`block_writer44`
** Message: Thread 4 dumping data for `test`.`block_writer45`
** Message: Thread 1 dumping data for `test`.`block_writer46`
** Message: Thread 2 dumping data for `test`.`block_writer47`
** Message: Thread 4 dumping data for `test`.`block_writer48`
** Message: Thread 3 dumping data for `test`.`block_writer49`
** Message: Thread 2 dumping data for `test`.`block_writer5`
** Message: Thread 4 dumping data for `test`.`block_writer50`
** Message: Thread 1 dumping data for `test`.`block_writer51`
** Message: Thread 3 dumping data for `test`.`block_writer52`
** Message: Thread 2 dumping data for `test`.`block_writer53`
** Message: Thread 2 dumping data for `test`.`block_writer54`
** Message: Thread 3 dumping data for `test`.`block_writer55`
** Message: Thread 4 dumping data for `test`.`block_writer56`
** Message: Thread 3 dumping data for `test`.`block_writer57`
** (mydumper:1): CRITICAL **: Error dumping table (test.block_writer51) data: other error: request to underhook engine failed
** Message: Empty table test.block_writer51
** Message: Thread 1 dumping data for `test`.`block_writer58`
** Message: Thread 2 dumping data for `test`.`block_writer59`
** Message: Thread 4 dumping data for `test`.`block_writer6`
** Message: Thread 3 dumping data for `test`.`block_writer60`
** Message: Thread 1 dumping data for `test`.`block_writer61`
** Message: Thread 2 dumping data for `test`.`block_writer62`
** Message: Thread 4 dumping data for `test`.`block_writer63`
** Message: Thread 3 dumping data for `test`.`block_writer7`
** Message: Thread 1 dumping data for `test`.`block_writer8`
** Message: Thread 1 dumping data for `test`.`block_writer9`
** Message: Thread 2 dumping data for `test`.`testTable`
** Message: Thread 2 dumping schema for `mysql`.`GLOBAL_VARIABLES`
** Message: Thread 2 dumping schema for `mysql`.`bind_info`
** Message: Thread 2 dumping schema for `mysql`.`columns_priv`
** Message: Thread 2 dumping schema for `mysql`.`db`
** Message: Thread 2 dumping schema for `mysql`.`default_roles`
** Message: Thread 2 dumping schema for `mysql`.`gc_delete_range`
** Message: Thread 2 dumping schema for `mysql`.`gc_delete_range_done`
** Message: Thread 2 dumping schema for `mysql`.`help_topic`
** Message: Thread 2 dumping schema for `mysql`.`role_edges`
** Message: Thread 2 dumping schema for `mysql`.`stats_buckets`
** Message: Thread 2 dumping schema for `mysql`.`stats_feedback`
** Message: Thread 2 dumping schema for `mysql`.`stats_histograms`
** Message: Thread 2 dumping schema for `mysql`.`stats_meta`
** Message: Thread 2 dumping schema for `mysql`.`stats_top_n`
** Message: Thread 2 dumping schema for `mysql`.`tables_priv`
** Message: Thread 2 dumping schema for `mysql`.`tidb`
** Message: Thread 2 dumping schema for `mysql`.`user`
** Message: Thread 2 dumping schema for `test`.`block_writer`
** Message: Thread 2 dumping schema for `test`.`block_writer1`
** Message: Thread 2 dumping schema for `test`.`block_writer10`
** Message: Thread 2 dumping schema for `test`.`block_writer11`
** Message: Thread 2 dumping schema for `test`.`block_writer12`
** Message: Thread 2 dumping schema for `test`.`block_writer13`
** Message: Thread 2 dumping schema for `test`.`block_writer14`
** Message: Thread 2 dumping schema for `test`.`block_writer15`
** Message: Thread 2 dumping schema for `test`.`block_writer16`
** Message: Thread 2 dumping schema for `test`.`block_writer17`
** Message: Thread 2 dumping schema for `test`.`block_writer18`
** Message: Thread 2 dumping schema for `test`.`block_writer19`
** Message: Thread 2 dumping schema for `test`.`block_writer2`
** Message: Thread 2 dumping schema for `test`.`block_writer20`
** Message: Thread 2 dumping schema for `test`.`block_writer21`
** Message: Thread 2 dumping schema for `test`.`block_writer22`
** Message: Thread 2 dumping schema for `test`.`block_writer23`
** Message: Thread 2 dumping schema for `test`.`block_writer24`
** Message: Thread 2 dumping schema for `test`.`block_writer25`
** Message: Thread 2 dumping schema for `test`.`block_writer26`
** Message: Thread 2 dumping schema for `test`.`block_writer27`
** Message: Thread 2 dumping schema for `test`.`block_writer28`
** Message: Thread 2 dumping schema for `test`.`block_writer29`
** Message: Thread 2 dumping schema for `test`.`block_writer3`
** Message: Thread 2 dumping schema for `test`.`block_writer30`
** Message: Thread 2 dumping schema for `test`.`block_writer31`
** Message: Thread 2 dumping schema for `test`.`block_writer32`
** Message: Thread 2 dumping schema for `test`.`block_writer33`
** Message: Thread 2 dumping schema for `test`.`block_writer34`
** Message: Thread 2 dumping schema for `test`.`block_writer35`
** Message: Thread 2 dumping schema for `test`.`block_writer36`
** Message: Thread 2 dumping schema for `test`.`block_writer37`
** Message: Thread 2 dumping schema for `test`.`block_writer38`
** Message: Thread 2 dumping schema for `test`.`block_writer39`
** Message: Thread 2 dumping schema for `test`.`block_writer4`
** Message: Thread 2 dumping schema for `test`.`block_writer40`
** Message: Thread 2 dumping schema for `test`.`block_writer41`
** Message: Thread 2 dumping schema for `test`.`block_writer42`
** Message: Thread 2 dumping schema for `test`.`block_writer43`
** Message: Thread 2 dumping schema for `test`.`block_writer44`
** Message: Thread 2 dumping schema for `test`.`block_writer45`
** Message: Thread 2 dumping schema for `test`.`block_writer46`
** Message: Thread 2 dumping schema for `test`.`block_writer47`
** Message: Thread 2 dumping schema for `test`.`block_writer48`
** Message: Thread 2 dumping schema for `test`.`block_writer49`
** Message: Thread 2 dumping schema for `test`.`block_writer5`
** Message: Thread 2 dumping schema for `test`.`block_writer50`
** Message: Thread 2 dumping schema for `test`.`block_writer51`
** Message: Thread 2 dumping schema for `test`.`block_writer52`
** Message: Thread 2 dumping schema for `test`.`block_writer53`
** Message: Thread 2 dumping schema for `test`.`block_writer54`
** Message: Thread 2 dumping schema for `test`.`block_writer55`
** Message: Thread 2 dumping schema for `test`.`block_writer56`
** Message: Thread 2 dumping schema for `test`.`block_writer57`
** Message: Thread 2 dumping schema for `test`.`block_writer58`
** Message: Thread 2 dumping schema for `test`.`block_writer59`
** Message: Thread 2 dumping schema for `test`.`block_writer6`
** Message: Thread 2 dumping schema for `test`.`block_writer60`
** Message: Thread 2 dumping schema for `test`.`block_writer61`
** Message: Thread 2 dumping schema for `test`.`block_writer62`
** Message: Thread 2 dumping schema for `test`.`block_writer63`
** Message: Thread 2 dumping schema for `test`.`block_writer7`
** Message: Thread 2 dumping schema for `test`.`block_writer8`
** Message: Thread 2 dumping schema for `test`.`block_writer9`
** Message: Thread 2 dumping schema for `test`.`testTable`
** Message: Thread 2 shutting down
** Message: Thread 3 shutting down
** Message: Thread 4 shutting down
** Message: Thread 1 shutting down
** Message: Finished dump at: 2019-05-29 06:27:50

What is changed and how it works?

Check List

Tests

E2E test
Stability test
No code

Code changes

Has Helm charts change

Side effects

Related changes

Need to cherry-pick to the release branch

Does this PR introduce a user-facing change?:

get TS and use it before full backup using mydumper

weekface · 2019-05-29T07:47:49Z

/run-e2e-tests

* set --verbose to 3: 0 = silent, 1 = errors, 2 = warnings, 3 = info

shuijing198799 · 2019-05-29T11:29:00Z

LGTM

weekface · 2019-05-30T10:24:47Z

/run-e2e-tests

* remove backup initContainers

weekface · 2019-05-30T11:35:17Z

/run-e2e-tests

…idb-operator into reduce-chunk-filesize

weekface · 2019-05-31T14:38:22Z

/run-e2e-tests

weekface · 2019-06-03T07:46:43Z

/run-e2e-tests

…idb-operator into reduce-chunk-filesize

weekface · 2019-06-04T06:36:46Z

/run-e2e-tests

…idb-operator into reduce-chunk-filesize

weekface · 2019-06-05T03:07:44Z

@xiaojingchen @aylei PTAL

xiaojingchen · 2019-06-05T03:21:43Z

/run-e2e-test

aylei

LGTM

weekface · 2019-06-05T07:37:41Z

/run-e2e-test

weekface requested review from xiaojingchen, tennix and shuijing198799 May 29, 2019 07:00

xiaojingchen previously approved these changes May 29, 2019

View reviewed changes

* reduce chunk-filesize to 10MB

1c82860

* set --verbose to 3: 0 = silent, 1 = errors, 2 = warnings, 3 = info

weekface dismissed xiaojingchen’s stale review via 1c82860 May 29, 2019 09:35

weekface force-pushed the reduce-chunk-filesize branch from b916747 to 1c82860 Compare May 29, 2019 09:35

weekface changed the title ~~reduce ad-hoc backup options --chunk-filesize from 100MB to 10MB~~ [DNM] reduce ad-hoc backup options --chunk-filesize from 100MB to 10MB May 29, 2019

shuijing198799 previously approved these changes May 29, 2019

View reviewed changes

weekface dismissed shuijing198799’s stale review via ce32a4d May 30, 2019 10:11

* fix stability backup test

bf9d17c

* remove backup initContainers

weekface force-pushed the reduce-chunk-filesize branch from 8a23e7d to bf9d17c Compare May 30, 2019 11:30

weekface changed the title ~~[DNM] reduce ad-hoc backup options --chunk-filesize from 100MB to 10MB~~ [DNM] reduce ad-hoc backup options --chunk-filesize from 100MB to 1MB May 30, 2019

weekface changed the title ~~[DNM] reduce ad-hoc backup options --chunk-filesize from 100MB to 1MB~~ [DNM] fix stability backup tests May 30, 2019

Merge branch 'master' into reduce-chunk-filesize

1b501ee

weekface added 2 commits May 31, 2019 19:54

fix binlog backup

22588c4

Merge branch 'reduce-chunk-filesize' of https://github.com/weekface/t…

bf29efe

…idb-operator into reduce-chunk-filesize

weekface changed the title ~~[DNM] fix stability backup tests~~ fix stability backup tests May 31, 2019

Merge branch 'master' into reduce-chunk-filesize

3e54391

weekface added 2 commits June 1, 2019 22:56

Merge branch 'master' into reduce-chunk-filesize

f08cbe7

tiny fix

f1d4720

weekface force-pushed the reduce-chunk-filesize branch from 585a741 to f1d4720 Compare June 1, 2019 15:07

weekface changed the title ~~fix stability backup tests~~ [DNM] fix stability backup tests Jun 1, 2019

weekface added 2 commits June 3, 2019 15:40

get TS and use it before full backup using mydumper

2b63bb0

Merge branch 'master' into reduce-chunk-filesize

affefab

weekface added 2 commits June 3, 2019 16:18

add comment

170d506

Merge branch 'reduce-chunk-filesize' of https://github.com/weekface/t…

9707106

…idb-operator into reduce-chunk-filesize

weekface changed the title ~~[DNM] fix stability backup tests~~ fix stability backup tests Jun 3, 2019

weekface changed the title ~~fix stability backup tests~~ get TS and use it before full backup using mydumper Jun 3, 2019

weekface added 4 commits June 4, 2019 10:38

tiny fix

a4a5aca

tiny fix

7a718ae

Merge branch 'reduce-chunk-filesize' of https://github.com/weekface/t…

84af786

…idb-operator into reduce-chunk-filesize

Merge branch 'master' into reduce-chunk-filesize

26a2649

weekface added 3 commits June 4, 2019 16:52

stash

ca09f00

Merge branch 'reduce-chunk-filesize' of https://github.com/weekface/t…

b3f8bce

…idb-operator into reduce-chunk-filesize

fix nil pointer dereference

4f76038

xiaojingchen approved these changes Jun 5, 2019

View reviewed changes

aylei approved these changes Jun 5, 2019

View reviewed changes

weekface merged commit ef33ba5 into pingcap:master Jun 5, 2019

weekface deleted the reduce-chunk-filesize branch June 5, 2019 09:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

get TS and use it before full backup using mydumper #534

get TS and use it before full backup using mydumper #534

weekface commented May 29, 2019 •

edited

Loading

weekface commented May 29, 2019

shuijing198799 commented May 29, 2019

weekface commented May 30, 2019

weekface commented May 30, 2019

weekface commented May 31, 2019

weekface commented Jun 3, 2019

weekface commented Jun 4, 2019

weekface commented Jun 5, 2019

xiaojingchen commented Jun 5, 2019

aylei left a comment

weekface commented Jun 5, 2019

get TS and use it before full backup using mydumper #534

get TS and use it before full backup using mydumper #534

Conversation

weekface commented May 29, 2019 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Does this PR introduce a user-facing change?:

weekface commented May 29, 2019

shuijing198799 commented May 29, 2019

weekface commented May 30, 2019

weekface commented May 30, 2019

weekface commented May 31, 2019

weekface commented Jun 3, 2019

weekface commented Jun 4, 2019

weekface commented Jun 5, 2019

xiaojingchen commented Jun 5, 2019

aylei left a comment

Choose a reason for hiding this comment

weekface commented Jun 5, 2019

weekface commented May 29, 2019 •

edited

Loading