Skip to content
This repository has been archived by the owner on Nov 24, 2023. It is now read-only.

test: try fix etcd error in CI #1773

Merged
merged 13 commits into from
Jun 23, 2021
Merged

test: try fix etcd error in CI #1773

merged 13 commits into from
Jun 23, 2021

Conversation

lance6716
Copy link
Collaborator

@lance6716 lance6716 commented Jun 15, 2021 •
edited
Loading

part of #1790

@ti-chi-bot ti-chi-bot added size/M and removed size/XS labels Jun 18, 2021
@lance6716
Copy link
Collaborator Author

lance6716 commented Jun 18, 2021 •
edited
Loading

fail to start embed etcd, RawCause: walpb: crc mismatch" has 12 lines of wait process dm-master.test exit..., so I suspect the kill -9 of waiting script isn't very well (may cause etcd data corruption)

and log doesn't contain closed etcd server

@glorv
Copy link
Collaborator

glorv commented Jun 18, 2021

/build

@glorv
Copy link
Collaborator

glorv commented Jun 18, 2021

/rebuild

@lance6716

This comment has been minimized.

Sign in to view
@lance6716 lance6716 changed the title Quality reinforcement (12): fix unstable test cases test: fix lost graceful kill in CI Jun 18, 2021
@lance6716
Copy link
Collaborator Author

/run-all-tests

@lance6716 lance6716 mentioned this pull request Jun 18, 2021
Closed
30 tasks
@lance6716 lance6716 changed the title test: fix lost graceful kill in CI [WIP] test: try fix etcd error "walpb: crc mismatch" Jun 18, 2021
@lance6716
Copy link
Collaborator Author

lance6716 commented Jun 19, 2021 •
edited
Loading

when "walpb: crc mismatch" error happens, I found that the etcd wal file contains many 0 bits inspected by xxd 🤔
image

@lance6716
Copy link
Collaborator Author

lance6716 commented Jun 19, 2021 •
edited
Loading

https://ci.pingcap.net/blue/organizations/jenkins/dm_ghpr_new_test/detail/dm_ghpr_new_test/688/pipeline

MySQL is stuck 🤔 cause: pingcap/dumpling#296

no etcd error this time

@lance6716
Copy link
Collaborator Author

lance6716 commented Jun 19, 2021 •
edited
Loading

/run-all-tests

https://ci.pingcap.net/blue/organizations/jenkins/dm_ghpr_new_test/detail/dm_ghpr_new_test/689/pipeline

no etcd error this time

@lance6716 lance6716 changed the title [WIP] test: try fix etcd error "walpb: crc mismatch" test: try fix etcd error in CI Jun 19, 2021
@lance6716 lance6716 added the needs-cherry-pick-release-2.0 This PR should be cherry-picked to release-2.0. Remove this label after cherry-picked to release-2.0 label Jun 19, 2021
@lance6716
Copy link
Collaborator Author

lance6716 commented Jun 19, 2021 •
edited
Loading

/run-all-tests

https://ci.pingcap.net/blue/organizations/jenkins/dm_ghpr_new_test/detail/dm_ghpr_new_test/690/pipeline

wal: max entry size limit exceeded

root@test-faa1678b-e502-4949-9aa9-9f15f2c3f387-wnpdm-21bqm:/tmp/dm_test/ha_master/master3/default.master3/member/wal# xxd -s -100 0000000000000000-0000000000000000.wal
03d08f9c: 0000 0000 0000 0000 0000 0000 0000 0000  ................
03d08fac: 0000 0000 0000 0000 0000 0000 0000 0000  ................
03d08fbc: 0000 0000 0000 0000 0000 0000 0000 0000  ................
03d08fcc: 0000 0000 0000 0000 0000 0000 0000 0000  ................
03d08fdc: 0000 0000 0000 0000 0000 0000 0000 0000  ................
03d08fec: 0000 0000 0000 0000 0000 0000 0000 0000  ................
03d08ffc: 0000 0000

@lance6716 lance6716 changed the title test: try fix etcd error in CI [WIP] test: try fix etcd error in CI Jun 19, 2021
@lance6716

This comment has been minimized.

Sign in to view
@ti-chi-bot ti-chi-bot added the status/LGT1 One reviewer already commented LGTM label Jun 21, 2021
@ti-chi-bot
Copy link
Member

@Ehco1996: Thanks for your review. The bot only counts approvals from reviewers and higher roles in list, but you're still welcome to leave your comments.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@lance6716
Copy link
Collaborator Author

Upgrade via TiUP / From V2 (v2.0.4) now passed

/run-all-tests

@lance6716
Copy link
Collaborator Author

port 8262 check offline timeout

will check the error later

@Ehco1996
Copy link
Contributor

@lance6716 @GMHDBJD @glorv @lichunzhu

ci passed , should we merge this PR first?

@lance6716
Copy link
Collaborator Author

/merge

@ti-chi-bot
Copy link
Member

@lance6716: /merge in this pull request requires 2 approval(s).

In response to this:

/merge

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

GMHDBJD
GMHDBJD approved these changes Jun 23, 2021
View reviewed changes
Copy link
Collaborator

@GMHDBJD GMHDBJD left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-chi-bot
Copy link
Member

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • GMHDBJD
  • glorv

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added status/LGT2 Two reviewers already commented LGTM, ready for merge and removed status/LGT1 One reviewer already commented LGTM labels Jun 23, 2021
@GMHDBJD
Copy link
Collaborator

GMHDBJD commented Jun 23, 2021

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 486525a

@ti-chi-bot ti-chi-bot merged commit 6c4d1ed into master Jun 23, 2021
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #1806.

ti-chi-bot pushed a commit to ti-chi-bot/dm that referenced this pull request Jun 23, 2021
@lance6716 @ti-chi-bot
test: try fix etcd error in CI (pingcap#1773)
ca8810d
@ti-chi-bot ti-chi-bot mentioned this pull request Jun 23, 2021
Merged
ti-chi-bot added a commit that referenced this pull request Jun 24, 2021
@ti-chi-bot
test: try fix etcd error in CI (#1773) (#1806)
442c206
@Ehco1996 Ehco1996 deleted the fix-CI-12 branch September 27, 2021 01:46
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@GMHDBJD GMHDBJD GMHDBJD approved these changes

@Ehco1996 Ehco1996 Ehco1996 approved these changes

@glorv glorv glorv approved these changes

@lichunzhu lichunzhu Awaiting requested review from lichunzhu

Assignees
No one assigned
Labels
needs-cherry-pick-release-2.0 This PR should be cherry-picked to release-2.0. Remove this label after cherry-picked to release-2.0 size/M status/can-merge status/LGT2 Two reviewers already commented LGTM, ready for merge
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@lance6716 @glorv @ti-chi-bot @Ehco1996 @GMHDBJD