Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Convert binlog.py to go version #41

Merged
merged 72 commits into from
Dec 18, 2019
Merged

Convert binlog.py to go version #41

merged 72 commits into from
Dec 18, 2019

Conversation

arindamnayak
Copy link

Ported binlog.py to go version.

arindamnayak and others added 30 commits October 31, 2019 12:17
* pick latest fix from cluster_for_test

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>
* tabletmanager lock unlock table test case

Signed-off-by: Ajeet jain <ajeet@planetscale.com>
* converted vtctld_test.py to go

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>
* migrated one of testcase from schema.py to schema_test.go

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>
* converted tabletmanager test cases to go

Signed-off-by: Ajeet jain <ajeet@planetscale.com>
* ported testcase of keyspace_test.py

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>
* Converted sharded test from py to go

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>
Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>
Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>
Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>
Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>
Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>
Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>
Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>
Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>
Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>
Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>
* pick latest fix from cluster_for_test

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>
* tabletmanager lock unlock table test case

Signed-off-by: Ajeet jain <ajeet@planetscale.com>
* converted vtctld_test.py to go

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>
* migrated one of testcase from schema.py to schema_test.go

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>
* converted tabletmanager test cases to go

Signed-off-by: Ajeet jain <ajeet@planetscale.com>
* ported testcase of keyspace_test.py

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>
* Converted sharded test from py to go

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>
Signed-off-by: saurabh <saurabh.408@gmail.com>
Signed-off-by: saurabh <saurabh.408@gmail.com>
Signed-off-by: saurabh <saurabh.408@gmail.com>
Signed-off-by: saurabh <saurabh.408@gmail.com>
Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>
Signed-off-by: saurabh <saurabh.408@gmail.com>
arindamnayak and others added 13 commits December 6, 2019 18:34
Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>
Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>
Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>
Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>
Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>
Signed-off-by: Ajeet jain <ajeet@planetscale.com>

* readme for go endtoend test cases

Signed-off-by: Ajeet jain <ajeet@planetscale.com>

* Update README.md
Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>
Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>
Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>
Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>
Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>
Copy link
Collaborator

@deepthi deepthi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you write a description for each test case similar to how you already have for TestCharset? Maybe also rename TestMain to TestBinlog? or whatever is appropriate.

Also requires a rebase/merge to get unit tests passing.

Comment on lines +254 to +259
// Insert something that will replicate incorrectly if the charset is not
// propagated through binlog streamer to the destination.

// Vitess tablets default to using utf8, so we insert something crazy and
// pretend it's latin1. If the binlog player doesn't also pretend it's
// latin1, it will be inserted as utf8, which will change its value.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. Nice documentation.

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>
@arindamnayak
Copy link
Author

Can you write a description for each test case similar to how you already have for TestCharset? Maybe also rename TestMain to TestBinlog? or whatever is appropriate.

Also requires a rebase/merge to get unit tests passing.

Update the documentations. TestMain contains all setup related code, we can't customize that name as it accepts m *testing.M .

It depends on a upstream PR (initial sharding PR), once that is merged and if this PR looks good to you, I will create PR against upstream, where all these unit test case failure will be fixed automatically.

Copy link
Collaborator

@deepthi deepthi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@arindamnayak arindamnayak changed the base branch from tal_test_initial_sharding_v3 to tal_test_master December 18, 2019 06:07
arindamnayak and others added 2 commits December 18, 2019 13:25
Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>
@ajeetj
Copy link

ajeetj commented Dec 18, 2019

LGTM, merging in tal_test_master

@ajeetj ajeetj merged commit 76d6f62 into tal_test_master Dec 18, 2019
arindamnayak added a commit that referenced this pull request Dec 18, 2019
Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>
Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>
ajeetj pushed a commit that referenced this pull request Dec 19, 2019
Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>
ajeetj added a commit that referenced this pull request Dec 20, 2019
* Rebase test_master with latest Cluster code (#22)

* pick latest fix from cluster_for_test

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* Tabletmanager2 test cases in GO using cluster (#23)

* tabletmanager lock unlock table test case

Signed-off-by: Ajeet jain <ajeet@planetscale.com>

* Vtctld Test cases in Go using cluster (#25)

* converted vtctld_test.py to go

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* Converted schema.py testcase (#26)

* migrated one of testcase from schema.py to schema_test.go

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* Tablet Manager test cases in Go using cluster (#27)

* converted tabletmanager test cases to go

Signed-off-by: Ajeet jain <ajeet@planetscale.com>

* Converted keyspace test to go (#28)

* ported testcase of keyspace_test.py

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* Converted sharded test from py to go (#29)

* Converted sharded test from py to go

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* initial commit for sharding test

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* added reconciliation assert

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* added insert and stream replication

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* added rest assertions including migrateservedtypes

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* completed remaining assertions and fixed non-workings

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* converted to v3 way of resharding

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* restructured the testcases

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* added testcase with bytes as sharding key type

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* Rebase test_master with latest Cluster code (#22)

* pick latest fix from cluster_for_test

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* Tabletmanager2 test cases in GO using cluster (#23)

* tabletmanager lock unlock table test case

Signed-off-by: Ajeet jain <ajeet@planetscale.com>

* Vtctld Test cases in Go using cluster (#25)

* converted vtctld_test.py to go

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* Converted schema.py testcase (#26)

* migrated one of testcase from schema.py to schema_test.go

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* Tablet Manager test cases in Go using cluster (#27)

* converted tabletmanager test cases to go

Signed-off-by: Ajeet jain <ajeet@planetscale.com>

* Converted keyspace test to go (#28)

* ported testcase of keyspace_test.py

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* Converted sharded test from py to go (#29)

* Converted sharded test from py to go

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* Added testcase for mysqlctl process

Signed-off-by: saurabh <saurabh.408@gmail.com>

* removed unused variables

Signed-off-by: saurabh <saurabh.408@gmail.com>

* removing comment

Signed-off-by: saurabh <saurabh.408@gmail.com>

* addressed review comments

Signed-off-by: saurabh <saurabh.408@gmail.com>

* removed cleandir flag from vttablet teardown

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* added v3 sharding test

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* vtgate buffer testcase migrated in go

Signed-off-by: saurabh <saurabh.408@gmail.com>

* inital code for resharding

Signed-off-by: Ajeet jain <ajeet@planetscale.com>

* completed the cluster code

Signed-off-by: Ajeet jain <ajeet@planetscale.com>

* more progress

Signed-off-by: Ajeet jain <ajeet@planetscale.com>

* pull graceful shutdown code

Signed-off-by: Ajeet jain <ajeet@planetscale.com>

* added fe more cases in resharding_test.go

Signed-off-by: Ajeet jain <ajeet@planetscale.com>

* fixed merge issues

Signed-off-by: Ajeet jain <ajeet@planetscale.com>

* running mysql in parallel mode and fe more assertions

Signed-off-by: Ajeet jain <ajeet@planetscale.com>

* used rbr mysql mode

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* almost finished with reshading v2 version

Signed-off-by: Ajeet jain <ajeet@planetscale.com>

* Rebase test_master with latest Cluster code (#22)

* pick latest fix from cluster_for_test

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* Tabletmanager2 test cases in GO using cluster (#23)

* tabletmanager lock unlock table test case

Signed-off-by: Ajeet jain <ajeet@planetscale.com>

* Vtctld Test cases in Go using cluster (#25)

* converted vtctld_test.py to go

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* Converted schema.py testcase (#26)

* migrated one of testcase from schema.py to schema_test.go

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* Tablet Manager test cases in Go using cluster (#27)

* converted tabletmanager test cases to go

Signed-off-by: Ajeet jain <ajeet@planetscale.com>

* Converted keyspace test to go (#28)

* ported testcase of keyspace_test.py

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* Converted sharded test from py to go (#29)

* Converted sharded test from py to go

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* Added testcase for mysqlctl process

Signed-off-by: saurabh <saurabh.408@gmail.com>

* removed unused variables

Signed-off-by: saurabh <saurabh.408@gmail.com>

* removing comment

Signed-off-by: saurabh <saurabh.408@gmail.com>

* addressed review comments

Signed-off-by: saurabh <saurabh.408@gmail.com>

* removed cleandir flag from vttablet teardown

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* vtgate buffer testcase migrated in go

Signed-off-by: saurabh <saurabh.408@gmail.com>

* changed code to use v3 way of sharding

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* converted to v3 way

Signed-off-by: Ajeet jain <ajeet@planetscale.com>

* added multi split diff and multi sharding

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* converted to v3 way

Signed-off-by: Ajeet jain <ajeet@planetscale.com>

* TabletManager remaining test cases

TabletManager remaining test cases in GO
Signed-off-by: Ajeet jain <ajeet@planetscale.com>

* moved sql start to non-blocking mode (#35)

* moved sql start to non-blocking mode

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* added testcase for vtgate vars

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* implement pr suggestion

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* trying to work with throttler

Signed-off-by: Ajeet jain <ajeet@planetscale.com>

* use ks1 mysql ports for other tablets in ks2

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* converted on int keys

Signed-off-by: Ajeet jain <ajeet@planetscale.com>

* Rebase test_master with latest Cluster code (#22)

* pick latest fix from cluster_for_test

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>
Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* Tabletmanager2 test cases in GO using cluster (#23)

* tabletmanager lock unlock table test case

Signed-off-by: Ajeet jain <ajeet@planetscale.com>

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* Vtctld Test cases in Go using cluster (#25)

* converted vtctld_test.py to go

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>
Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* Converted schema.py testcase (#26)

* migrated one of testcase from schema.py to schema_test.go

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>
Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* Tablet Manager test cases in Go using cluster (#27)

* converted tabletmanager test cases to go

Signed-off-by: Ajeet jain <ajeet@planetscale.com>

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* Converted keyspace test to go (#28)

* ported testcase of keyspace_test.py

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>
Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* Converted sharded test from py to go (#29)

* Converted sharded test from py to go

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>
Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* Added testcase for mysqlctl process

Signed-off-by: saurabh <saurabh.408@gmail.com>
Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* removed unused variables

Signed-off-by: saurabh <saurabh.408@gmail.com>
Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* removing comment

Signed-off-by: saurabh <saurabh.408@gmail.com>
Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* addressed review comments

Signed-off-by: saurabh <saurabh.408@gmail.com>
Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* removed cleandir flag from vttablet teardown

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* vtgate buffer testcase migrated in go

Signed-off-by: saurabh <saurabh.408@gmail.com>
Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* TabletManager remaining test cases

TabletManager remaining test cases in GO
Signed-off-by: Ajeet jain <ajeet@planetscale.com>
Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* moved sql start to non-blocking mode (#35)

* moved sql start to non-blocking mode

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>
Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* added testcase for vtgate vars

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* redirect process errors to log files (#38)

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* debug the test error in ci

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* fix initial sharding multi

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* fixed protected mysqlctl teardown

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* removed sleep time

Signed-off-by: Ajeet jain <ajeet@planetscale.com>

* refactor mysqlctl start and stop

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* Fixed multisplitdiff command

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* added log statement for error condition

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* Ignoring Python tests which are moved to GO (#40)

Signed-off-by: Ajeet jain <ajeet@planetscale.com>

* readme for go endtoend test cases

Signed-off-by: Ajeet jain <ajeet@planetscale.com>

* Update README.md

* Fixed byte column test cases

Signed-off-by: Ajeet jain <ajeet@planetscale.com>

* lint fixes

Signed-off-by: Ajeet jain <ajeet@planetscale.com>

* updated comments

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* restructure of launch cluster

Signed-off-by: Ajeet jain <ajeet@planetscale.com>

* fix tear down code

Signed-off-by: Ajeet jain <ajeet@planetscale.com>

* Refactor code

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* fix the throttler test cases

Signed-off-by: Ajeet jain <ajeet@planetscale.com>

* fix the static checks

Signed-off-by: Ajeet jain <ajeet@planetscale.com>

* disable running python test cases in travis

Signed-off-by: Ajeet jain <ajeet@planetscale.com>

* fixed static checks

Signed-off-by: Ajeet jain <ajeet@planetscale.com>

* removed unrequired line to set RBR mode

Signed-off-by: Ajeet jain <ajeet@planetscale.com>

* Add matrix build for mysql

Signed-off-by: Morgan Tocker <tocker@gmail.com>

* Add additional flavors, move flaky test

Signed-off-by: Morgan Tocker <tocker@gmail.com>

* vindex: add required NeedVCursor function

There's currently no way to know if a vindex requires the vcursor.
We need this knowledge to prevent vreplication from choosing
a vindex that requires a VCursor. This change requires all vindexes
to report if they need the vcursor or not.

Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>

* vrepl: choose best vindex instead of primary

Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>

* few of the reparent test cases

Signed-off-by: Ajeet jain <ajeet@planetscale.com>

* vttablet-up.sh : Adds detection of mysqld launch failure.  Given failure, tests for further for          presence of apparmor.  Exits with error and recommendation to disable.

Signed-off-by: ChrisR <radsurfer@gmail.com>

* completed all the reparent tests

Signed-off-by: Ajeet jain <ajeet@planetscale.com>

* Remove MariaDB 10.1

Re-enable unit race
Change CI git checkout to be v2.0
Disable test caching

Signed-off-by: Morgan Tocker <tocker@gmail.com>

* Revert some of the MariaDB changes, fix unit_race

Signed-off-by: Morgan Tocker <tocker@gmail.com>

* Add debugging for macOS issue

Signed-off-by: Morgan Tocker <tocker@gmail.com>

* Add additional flaky test

Signed-off-by: Morgan Tocker <tocker@gmail.com>

* Add sleep for macOS

Signed-off-by: Morgan Tocker <tocker@gmail.com>

* Disable macOS local example for now (flaky)

Signed-off-by: Morgan Tocker <tocker@gmail.com>

* Disable unit_race again

Signed-off-by: Morgan Tocker <tocker@gmail.com>

* updated test cases for reparent

Signed-off-by: Ajeet jain <ajeet@planetscale.com>

* Resharding endtoend testcases in Go migrated from Python (#44)


Signed-off-by: Ajeet jain <ajeet@planetscale.com>

* fix vtgate schema testcases

Signed-off-by: Ajeet jain <ajeet@planetscale.com>

* Convert binlog.py to go version (#41)


Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* fixed range based reparenting

Signed-off-by: Ajeet jain <ajeet@planetscale.com>

* removed unrequired comments

Signed-off-by: Ajeet jain <ajeet@planetscale.com>

* Modified apparmor detection to passive, added additional investigation info.

Signed-off-by: ChrisR <radsurfer@gmail.com>

* make-release-packages.sh: Disable shellcheck 1091 and Add README pointing to local install instructions

Signed-off-by: Gary Edgar <gary@planetscale.com>

* review comments addressed

Signed-off-by: Ajeet jain <ajeet@planetscale.com>

* Splitting github endtoend action into 2 files

Signed-off-by: Ajeet jain <ajeet@planetscale.com>

* vindex: NeedVCursor->NeedsVCursor

Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>

* Set correct tablet type to fix failing test

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* Fixing initial sharding multi

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* fix the assertions

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* fixed master-spare test case

Signed-off-by: Ajeet jain <ajeet@planetscale.com>

* Testing dockerized test in github workflow

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* Vertical split testcase migrated in go (#46)

Vertical split endtoend testcases in Go migrated from Python

Signed-off-by: saurabh <saurabh.408@gmail.com>

* Adjusted shard number for testcase

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* Reverting changes for config.json

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

* Refactor vitess/lite Dockerfiles. (vitessio#5592)

Signed-off-by: Anthony Yeh <enisoc@planetscale.com>

Co-authored-by: Arindam Nayak <arindamnayak@users.noreply.github.com>
Co-authored-by: saurabh408 <saurabh.408@gmail.com>
Co-authored-by: Morgan Tocker <tocker@gmail.com>
Co-authored-by: Sugu Sougoumarane <ssougou@gmail.com>
Co-authored-by: ChrisR <radsurfer@gmail.com>
Co-authored-by: Gary Edgar <gary@planetscale.com>
Co-authored-by: Anthony Yeh <enisoc@planetscale.com>
Co-authored-by: Deepthi Sigireddi <deepthi.sigireddi@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants