Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TestResourceManagerClientFailover is unstable #7400

Closed
HuSharp opened this issue Nov 21, 2023 · 1 comment · Fixed by #7402
Closed

TestResourceManagerClientFailover is unstable #7400

HuSharp opened this issue Nov 21, 2023 · 1 comment · Fixed by #7402
Labels
type/ci The issue is related to CI.

Comments

@HuSharp
Copy link
Member

HuSharp commented Nov 21, 2023

Flaky Test

TestResourceManagerClientFailover

Which jobs are failing

resource_manager_test.go:173: 
2023-11-21T04:55:18.4440291Z         	Error Trace:	/home/runner/work/pd/pd/tests/integrations/mcs/resourcemanager/resource_manager_test.go:173
2023-11-21T04:55:18.4441373Z         	            				/home/runner/work/pd/pd/tests/integrations/mcs/resourcemanager/resource_manager_test.go:1085
2023-11-21T04:55:18.4441600Z         	Error:      	Received unexpected error:
2023-11-21T04:55:18.4442831Z         	            	[PD:etcd:ErrEtcdMoveLeader]etcdserver: request timed out, leader transfer took too long: etcdserver: request timed out, leader transfer took too long
2023-11-21T04:55:18.4443176Z         	            	github.com/pingcap/errors.AddStack
2023-11-21T04:55:18.4444369Z         	            		/home/runner/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/errors.go:174
2023-11-21T04:55:18.4444838Z         	            	github.com/pingcap/errors.(*Error).GenWithStackByCause
2023-11-21T04:55:18.4445931Z         	            		/home/runner/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/normalize.go:307
2023-11-21T04:55:18.4446649Z         	            	github.com/tikv/pd/pkg/member.(*EmbeddedEtcdMember).MoveEtcdLeader
2023-11-21T04:55:18.4447142Z         	            		/home/runner/work/pd/pd/pkg/member/member.go:314
2023-11-21T04:55:18.4447732Z         	            	github.com/tikv/pd/pkg/member.(*EmbeddedEtcdMember).ResignEtcdLeader
2023-11-21T04:55:18.4448206Z         	            		/home/runner/work/pd/pd/pkg/member/member.go:376
2023-11-21T04:55:18.4448644Z         	            	github.com/tikv/pd/tests.(*TestServer).ResignLeader
2023-11-21T04:55:18.4449085Z         	            		/home/runner/work/pd/pd/tests/cluster.go:170
2023-11-21T04:55:18.4449529Z         	            	github.com/tikv/pd/tests.(*TestCluster).ResignLeader
2023-11-21T04:55:18.4449968Z         	            		/home/runner/work/pd/pd/tests/cluster.go:711
2023-11-21T04:55:18.4451000Z         	            	github.com/tikv/pd/tests/integrations/mcs/resourcemanager_test.(*resourceManagerClientTestSuite).resignAndWaitLeader
2023-11-21T04:55:18.4451908Z         	            		/home/runner/work/pd/pd/tests/integrations/mcs/resourcemanager/resource_manager_test.go:173
2023-11-21T04:55:18.4453076Z         	            	github.com/tikv/pd/tests/integrations/mcs/resourcemanager_test.(*resourceManagerClientTestSuite).TestResourceManagerClientFailover
2023-11-21T04:55:18.4453980Z         	            		/home/runner/work/pd/pd/tests/integrations/mcs/resourcemanager/resource_manager_test.go:1085
2023-11-21T04:55:18.4454224Z         	            	reflect.Value.call
2023-11-21T04:55:18.4454799Z         	            		/opt/hostedtoolcache/go/1.21.4/x64/src/reflect/value.go:596
2023-11-21T04:55:18.4455041Z         	            	reflect.Value.Call
2023-11-21T04:55:18.4455604Z         	            		/opt/hostedtoolcache/go/1.21.4/x64/src/reflect/value.go:380
2023-11-21T04:55:18.4455995Z         	            	github.com/stretchr/testify/suite.Run.func1
2023-11-21T04:55:18.4456724Z         	            		/home/runner/go/pkg/mod/github.com/stretchr/testify@v1.8.2/suite/suite.go:197
2023-11-21T04:55:18.4456955Z         	            	testing.tRunner
2023-11-21T04:55:18.4457542Z         	            		/opt/hostedtoolcache/go/1.21.4/x64/src/testing/testing.go:1595
2023-11-21T04:55:18.4457776Z         	            	runtime.goexit
2023-11-21T04:55:18.4458321Z         	            		/opt/hostedtoolcache/go/1.21.4/x64/src/runtime/asm_amd64.s:1650
2023-11-21T04:55:18.4458774Z         	Test:       	TestResourceManagerClientTestSuite/TestResourceManagerClientFailover

CI link

https://github.com/tikv/pd/actions/runs/6939184278/job/18876691105?pr=7398

Reason for failure (if possible)

Anything else
@HuSharp HuSharp added the type/ci The issue is related to CI. label Nov 21, 2023
@lhy1024
Copy link
Contributor

lhy1024 commented Nov 21, 2023

2023-11-21T04:55:18.4388339Z [2023/11/21 04:54:47.563 +00:00] [WARN] [member.go:189] ["campaign times is too frequent, resign and campaign again"] [leader-name=pd2] [leader-key=/pd/7303774118822659565/leader]
It is the similiar with #7381

@lhy1024 lhy1024 mentioned this issue Nov 21, 2023
Merged
ti-chi-bot bot added a commit that referenced this issue Nov 22, 2023
@lhy1024 @ti-chi-bot
*: make resign leader test more stable (#7402)
20da1c7 
close #7400

Signed-off-by: lhy1024 <admin@liudos.us>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
rleungx pushed a commit to rleungx/pd that referenced this issue Dec 1, 2023
@lhy1024 @ti-chi-bot @rleungx
*: make resign leader test more stable (tikv#7402)
e6aebe9 
close tikv#7400

Signed-off-by: lhy1024 <admin@liudos.us>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/ci The issue is related to CI.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

*: make resign leader test more stable lhy1024/pd
2 participants
@lhy1024 @HuSharp