Skip to content
This repository has been archived by the owner on Aug 21, 2023. It is now read-only.

tests: add e2e test #34

Merged
merged 18 commits into from
Mar 10, 2020
Merged

tests: add e2e test #34

merged 18 commits into from
Mar 10, 2020

Conversation

3pointer
Copy link
Collaborator

@3pointer 3pointer commented Mar 4, 2020 •
edited
Loading

try to resolve #3, add an e2e test for dumpling.

  1. workload: MySQL -> dumpling -> lightning -> TiDB
  2. use sync_diff_inspector check data in MySQL and TiDB

@3pointer 3pointer added the WIP label Mar 4, 2020
@3pointer 3pointer requested a review from kennytm March 4, 2020 10:54
kennytm
kennytm reviewed Mar 4, 2020
View reviewed changes
tests/_utils/check_sync_diff Outdated Show resolved Hide resolved
tests/_utils/check_sync_diff Show resolved Hide resolved
kennytm
kennytm reviewed Mar 5, 2020
View reviewed changes
install.sh Outdated Show resolved Hide resolved
IANTHEREAL
IANTHEREAL reviewed Mar 5, 2020
View reviewed changes
tests/_utils/run_services Outdated
sleep 3
done

echo "Starting PD..."
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does mocktikv tidb need pd?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's lightning needs pd to start

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can turn off --check-requirements

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

okay, let me try

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when I removed pd-server, it seems lightning will try to get settings from tidb, then it will get this error:

[error="cannot decode settings from TiDB, please manually fill in `tidb.port` and `tidb.pd-addr`: json: cannot unmarshal bool into Go struct field Log.log.enable-slow-log of type uint32"]

seems tidb has changed log.EnableSlowLog from uint32 to bool in master
we need update tidb dependency in lightning first.
then use latest lightining

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, please file an issue to Lightning to track it.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure,have filed an issue

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe I can keep pd-server here in this PR, leave TODO to remove it in the future. @kennytm after lightning fix the issue, I'll use another PR to fix it.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found a proper version (v3.1.0.beta.1) that has no such problem. CI fixed PTAL

3pointer
3pointer commented Mar 6, 2020
View reviewed changes
tests/file_size/run.sh
@@ -10,7 +10,7 @@ run_sql "create table t (a varchar(255))"
chars_20="1111_0000_1111_0000_"

# insert 100 records, each occupies 20 bytes
run_sql "insert into t values $(seq -s, 100 | sed "s/[0-9]\+/('$chars_20')/g")"
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this shell can not run on mac OS, so change it

@IANTHEREAL
Copy link
Collaborator

please remove fix.sql

IANTHEREAL
IANTHEREAL reviewed Mar 10, 2020
View reviewed changes
install.sh

# FIXME: change to latest version after lightning fix issue
# https://github.com/pingcap/tidb-lightning/issues/277
TAG="v3.1.0-beta.1"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it better to always use latest lightning?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

an available lightning is enough, this test aimed to test dumpling not lightning

@IANTHEREAL
Copy link
Collaborator

Rest LGTM

@3pointer @IANTHEREAL
Update tests/_utils/check_sync_diff
c969d17 
Co-Authored-By: Ian <ArGregoryIan@gmail.com>
@kennytm kennytm added status/LGT1 One reviewer approved (LGTM1) and removed status/PTAL labels Mar 10, 2020
@kennytm kennytm merged commit 6828e6a into pingcap:master Mar 10, 2020
tisonkun pushed a commit to tisonkun/dumpling that referenced this pull request Oct 20, 2021
@3pointer @kennytm @IANTHEREAL
tests: add e2e test (pingcap#34)
4873cec 
* tests: add e2e test

* fix ci

* Update tests/_utils/check_sync_diff

Co-Authored-By: kennytm <kennytm@gmail.com>

* fix ci

* fix ci

* remove pd-server

* fix ci

* fix ci

* fix ci

* fix ci

* debug ci

* debug ci

* fix ci

* leave TODO

* Update tests/_utils/check_sync_diff

Co-Authored-By: Ian <ArGregoryIan@gmail.com>

Co-authored-by: kennytm <kennytm@gmail.com>
Co-authored-by: Ian <ArGregoryIan@gmail.com>
tisonkun pushed a commit to tisonkun/dumpling that referenced this pull request Oct 20, 2021
@3pointer @kennytm @IANTHEREAL
tests: add e2e test (pingcap#34)
3b388b4 
* tests: add e2e test

* fix ci

* Update tests/_utils/check_sync_diff

Co-Authored-By: kennytm <kennytm@gmail.com>

* fix ci

* fix ci

* remove pd-server

* fix ci

* fix ci

* fix ci

* fix ci

* debug ci

* debug ci

* fix ci

* leave TODO

* Update tests/_utils/check_sync_diff

Co-Authored-By: Ian <ArGregoryIan@gmail.com>

Co-authored-by: kennytm <kennytm@gmail.com>
Co-authored-by: Ian <ArGregoryIan@gmail.com>
tisonkun pushed a commit to tisonkun/dumpling that referenced this pull request Oct 20, 2021
@3pointer @kennytm @IANTHEREAL
tests: add e2e test (pingcap#34)
7f5e371 
* tests: add e2e test

* fix ci

* Update tests/_utils/check_sync_diff

Co-Authored-By: kennytm <kennytm@gmail.com>

* fix ci

* fix ci

* remove pd-server

* fix ci

* fix ci

* fix ci

* fix ci

* debug ci

* debug ci

* fix ci

* leave TODO

* Update tests/_utils/check_sync_diff

Co-Authored-By: Ian <ArGregoryIan@gmail.com>

Co-authored-by: kennytm <kennytm@gmail.com>
Co-authored-by: Ian <ArGregoryIan@gmail.com>
tisonkun pushed a commit to tisonkun/dumpling that referenced this pull request Oct 20, 2021
@3pointer @kennytm @IANTHEREAL
tests: add e2e test (pingcap#34)
fced0e4 
* tests: add e2e test

* fix ci

* Update tests/_utils/check_sync_diff

Co-Authored-By: kennytm <kennytm@gmail.com>

* fix ci

* fix ci

* remove pd-server

* fix ci

* fix ci

* fix ci

* fix ci

* debug ci

* debug ci

* fix ci

* leave TODO

* Update tests/_utils/check_sync_diff

Co-Authored-By: Ian <ArGregoryIan@gmail.com>

Co-authored-by: kennytm <kennytm@gmail.com>
Co-authored-by: Ian <ArGregoryIan@gmail.com>
tisonkun pushed a commit to tisonkun/tidb that referenced this pull request Oct 20, 2021
@3pointer @kennytm
@IANTHEREAL @tisonkun
tests: add e2e test (pingcap/dumpling#34)
ff0bdbc 
* tests: add e2e test

* fix ci

* Update tests/_utils/check_sync_diff

Co-Authored-By: kennytm <kennytm@gmail.com>

* fix ci

* fix ci

* remove pd-server

* fix ci

* fix ci

* fix ci

* fix ci

* debug ci

* debug ci

* fix ci

* leave TODO

* Update tests/_utils/check_sync_diff

Co-Authored-By: Ian <ArGregoryIan@gmail.com>

Co-authored-by: kennytm <kennytm@gmail.com>
Co-authored-by: Ian <ArGregoryIan@gmail.com>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@IANTHEREAL IANTHEREAL IANTHEREAL left review comments

@kennytm kennytm kennytm approved these changes

Assignees
No one assigned
Labels
status/LGT1 One reviewer approved (LGTM1)
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

E2E test
3 participants
@3pointer @IANTHEREAL @kennytm