add failpoint test for db operation #206

WangXiangUSTC · 2019-07-18T08:42:00Z

What problem does this PR solve?

add some failpoint about db operation, and add integration test
issue: https://internal.pingcap.net/jira/browse/TOOL-1394

What is changed and how it works?

add some fail point about db operation
use these fail point to add integration test

initial unit failed, and find a bug which will lead to panic on dm-worker and dm-master
start task failed
relay log failed

Check List

Tests

Integration test

codecov · 2019-07-19T02:14:17Z

Codecov Report

Merging #206 into master will not change coverage.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             master       #206   +/-   ##
===========================================
  Coverage   58.8389%   58.8389%           
===========================================
  Files           123        123           
  Lines         14487      14487           
===========================================
  Hits           8524       8524           
  Misses         5098       5098           
  Partials        865        865

amyangfei

Note the integration test runs in parallel, and new added test case is not listed in CI, so remember to add the TEST_NAME of new added test case to file others_integration.txt in a newline.

Newly added integration tests are not tested now

…ang/db_test

WangXiangUSTC · 2019-07-24T06:22:54Z

/run-all-tests

WangXiangUSTC · 2019-07-24T06:42:23Z

/run-all-tests

WangXiangUSTC · 2019-07-24T08:04:32Z

/run-all-tests

dm/worker/subtask.go

WangXiangUSTC · 2019-07-24T08:15:44Z

/run-all-tests

WangXiangUSTC · 2019-07-24T09:41:35Z

@amyangfei have already add new test in others_integration.txt

IANTHEREAL · 2019-07-29T05:41:05Z

loader/db.go

+
+		failpoint.Inject("LoadExecCreateTableFailed", func() {
+			if i == 0 && len(sqls) == 1 && strings.Contains(sqls[0], "CREATE TABLE") {
+				err = domain.ErrInfoSchemaChanged


why only inject ErrInfoSchemaChanged error here? can we support to inject custom error?

use error code, 3b35795

IANTHEREAL · 2019-07-29T06:04:48Z

tests/load_interrupt/run.sh

@@ -59,7 +59,8 @@ function run() {
    check_row_count 1
    check_row_count 2

-    export GO_FAILPOINTS=''
+    # only failed at the first time, will retry later and success
+    export GO_FAILPOINTS='github.com/pingcap/dm/loader/LoadExecCreateTableFailed=return(true)'


how to check it?

add log check

# LoadExecCreateTableFailed error return twice err_cnt=`grep LoadExecCreateTableFailed $WORK_DIR/worker1/log/dm-worker.log | wc -l` if [ $err_cnt -ne 2 ]; then echo "error LoadExecCreateTableFailed's count is not 2" exit 2 fi

IANTHEREAL · 2019-07-29T06:07:02Z

Good Job！

…ang/db_test

WangXiangUSTC · 2019-07-30T04:46:15Z

/run-all-tests

WangXiangUSTC · 2019-07-30T04:55:55Z

@GregoryIan @amyangfei PTAL again

loader/db.go

amyangfei · 2019-07-31T02:52:16Z

tests/_utils/test_prepare

@@ -25,6 +25,15 @@ function cleanup2() {
    pkill -hup dm-tracer.test 2>/dev/null || true
 }

+function cleanup() {


I'm afraid that cleanup cleans more data then what we need, such as it cleans running dir after a test case finished, which is not friendly for error inspection.
What about moving wait_process_exit to cleanup2, and using cleanup1 or cleanup2 when we need. Besides we can use a better name then cleanup1, cleanup2

update in 9d7936a

pkg/utils/common.go

pkg/utils/db.go

Co-Authored-By: amyangfei <amyangfei@gmail.com>

amyangfei · 2019-07-31T08:25:39Z

/run-all-tests

amyangfei

LGTM

WangXiangUSTC · 2019-07-31T08:34:22Z

@GregoryIan PTAL again

IANTHEREAL · 2019-08-01T05:17:35Z

/run-all-tests

IANTHEREAL · 2019-08-01T05:21:25Z

/run-all-tests

IANTHEREAL

LGTM

WangXiangUSTC added 3 commits July 17, 2019 17:01

add failpoint for loader when execute create table

09dd4bd

enable failpoint in test

0eaceba

minor update

e6e2a2d

WangXiangUSTC added the status/WIP This PR is still work in progress label Jul 18, 2019

fix check

9a606fc

amyangfei added priority/normal Minor change, requires approval from ≥1 primary reviewer type/qa relate to quality assurance labels Jul 19, 2019

WangXiangUSTC added 5 commits July 22, 2019 17:26

add failpoint

580ee08

add test case

0e673c6

add test case for relay failed

c9b0b77

refine test

155710f

clean code

fbf8a16

WangXiangUSTC added status/PTAL This PR is ready for review. Add this label back after committing new changes and removed status/WIP This PR is still work in progress labels Jul 24, 2019

WangXiangUSTC marked this pull request as ready for review July 24, 2019 05:29

Merge branch 'master' into xiang/db_test

6d65cf3

amyangfei reviewed Jul 24, 2019

View reviewed changes

WangXiangUSTC added 2 commits July 24, 2019 14:03

update others_integration.txt

738bcd5

Merge branch 'xiang/db_test' of https://github.com/pingcap/dm into xi…

cd9d793

…ang/db_test

WangXiangUSTC added 2 commits July 24, 2019 15:30

fix test

b688cb2

fix test

74c4f97

kennytm reviewed Jul 24, 2019

View reviewed changes

dm/worker/subtask.go Outdated Show resolved Hide resolved

WangXiangUSTC added 2 commits July 25, 2019 13:39

address commnet

5165e14

minor fix

fcf20ba

WangXiangUSTC added the priority/important Major change, requires approval from ≥2 primary reviewers label Jul 26, 2019

IANTHEREAL reviewed Jul 29, 2019

View reviewed changes

Merge branch 'master' into xiang/db_test

cc04334

WangXiangUSTC added 3 commits July 30, 2019 12:26

address comment

3b35795

Merge branch 'xiang/db_test' of https://github.com/pingcap/dm into xi…

0eb7170

…ang/db_test

fix

44d5641

Merge branch 'master' into xiang/db_test

3169dab

amyangfei reviewed Jul 31, 2019

View reviewed changes

WangXiangUSTC and others added 4 commits July 31, 2019 11:40

Update pkg/utils/db.go

f405bd1

Co-Authored-By: amyangfei <amyangfei@gmail.com>

Update loader/db.go

dfaf6cc

Co-Authored-By: amyangfei <amyangfei@gmail.com>

change cleanup1 to cleanup_data, change cleanup2 to cleanup_process

9d7936a

minor fix

84d41a0

amyangfei reviewed Jul 31, 2019

View reviewed changes

amyangfei added status/LGT1 One reviewer already commented LGTM and removed status/PTAL This PR is ready for review. Add this label back after committing new changes labels Jul 31, 2019

Merge branch 'master' into xiang/db_test

9e3900d

Merge branch 'master' into xiang/db_test

05ac82e

IANTHEREAL approved these changes Aug 1, 2019

View reviewed changes

IANTHEREAL merged commit ffebdd3 into master Aug 1, 2019

IANTHEREAL deleted the xiang/db_test branch August 1, 2019 06:11

amyangfei mentioned this pull request Aug 1, 2019

increase DM unit test/integration test code coverage #70

Open

22 tasks

lichunzhu pushed a commit to lichunzhu/dm that referenced this pull request Apr 6, 2020

add failpoint test for db operation (pingcap#206)

63671dc

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add failpoint test for db operation #206

add failpoint test for db operation #206

WangXiangUSTC commented Jul 18, 2019 •

edited

Loading

codecov bot commented Jul 19, 2019 •

edited

Loading

amyangfei left a comment

WangXiangUSTC commented Jul 24, 2019

WangXiangUSTC commented Jul 24, 2019

WangXiangUSTC commented Jul 24, 2019

WangXiangUSTC commented Jul 24, 2019

WangXiangUSTC commented Jul 24, 2019

IANTHEREAL Jul 29, 2019

WangXiangUSTC Jul 30, 2019

IANTHEREAL Jul 29, 2019

WangXiangUSTC Jul 30, 2019

IANTHEREAL commented Jul 29, 2019

WangXiangUSTC commented Jul 30, 2019

WangXiangUSTC commented Jul 30, 2019

amyangfei Jul 31, 2019 •

edited

Loading

WangXiangUSTC Jul 31, 2019

amyangfei commented Jul 31, 2019

amyangfei left a comment

WangXiangUSTC commented Jul 31, 2019

IANTHEREAL commented Aug 1, 2019

IANTHEREAL commented Aug 1, 2019

IANTHEREAL left a comment

add failpoint test for db operation #206

add failpoint test for db operation #206

Conversation

WangXiangUSTC commented Jul 18, 2019 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

codecov bot commented Jul 19, 2019 • edited Loading

Codecov Report

amyangfei left a comment

Choose a reason for hiding this comment

WangXiangUSTC commented Jul 24, 2019

WangXiangUSTC commented Jul 24, 2019

WangXiangUSTC commented Jul 24, 2019

WangXiangUSTC commented Jul 24, 2019

WangXiangUSTC commented Jul 24, 2019

IANTHEREAL Jul 29, 2019

Choose a reason for hiding this comment

WangXiangUSTC Jul 30, 2019

Choose a reason for hiding this comment

IANTHEREAL Jul 29, 2019

Choose a reason for hiding this comment

WangXiangUSTC Jul 30, 2019

Choose a reason for hiding this comment

IANTHEREAL commented Jul 29, 2019

WangXiangUSTC commented Jul 30, 2019

WangXiangUSTC commented Jul 30, 2019

amyangfei Jul 31, 2019 • edited Loading

Choose a reason for hiding this comment

WangXiangUSTC Jul 31, 2019

Choose a reason for hiding this comment

amyangfei commented Jul 31, 2019

amyangfei left a comment

Choose a reason for hiding this comment

WangXiangUSTC commented Jul 31, 2019

IANTHEREAL commented Aug 1, 2019

IANTHEREAL commented Aug 1, 2019

IANTHEREAL left a comment

Choose a reason for hiding this comment

WangXiangUSTC commented Jul 18, 2019 •

edited

Loading

codecov bot commented Jul 19, 2019 •

edited

Loading

amyangfei Jul 31, 2019 •

edited

Loading