Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TestHTTPClientTestSuiteWithServiceDiscovery is unstable #8555

Closed
lhy1024 opened this issue Aug 22, 2024 · 1 comment · Fixed by #8556
Closed

TestHTTPClientTestSuiteWithServiceDiscovery is unstable #8555

lhy1024 opened this issue Aug 22, 2024 · 1 comment · Fixed by #8556
Labels
type/ci The issue is related to CI.

Comments

@lhy1024
Copy link
Contributor

lhy1024 commented Aug 22, 2024

Flaky Test

Which jobs are failing

FAIL: TestHTTPClientTestSuiteWithServiceDiscovery/TestTransferLeader

CI link

https://github.com/tikv/pd/actions/runs/10487662345/job/29048498057

2024-08-21T09:59:37.1850214Z     http_client_test.go:637: 
2024-08-21T09:59:37.1851322Z         	Error Trace:	/home/runner/work/pd/pd/tests/integrations/client/http_client_test.go:637
2024-08-21T09:59:37.1852262Z         	Error:      	Should not be: "pd1"
2024-08-21T09:59:37.1853014Z         	Test:       	TestHTTPClientTestSuiteWithServiceDiscovery/TestTransferLeader

Reason for failure (if possible)

Anything else

@lhy1024 lhy1024 added the type/ci The issue is related to CI. label Aug 22, 2024
@HuSharp
Copy link
Member

HuSharp commented Aug 22, 2024

We can find PD2 campaign times by log:

  1. SetupSuite make PD2 as leader ----- 1

  2. TestRetryOnLeaderChange

TestRetryOnLeaderChange is before TestTransferLeader, which have 3 times resign

Pd2->PD1
Pd1->PD2 -------- 2
Pd2->PD1 --------- during this resign etcd had stuck which make PD2 still can start campaign leader

This is the reason why the test is unstable, if it's not stuck, for the whole test, here the campaign times won't +1

2024-08-21T09:59:37.1252137Z [2024/08/21 09:59:21.828 +00:00] [INFO] [member.go:362] ["try to resign etcd leader to next pd-server"] [from=pd2] [to=]
2024-08-21T09:59:37.1253432Z [2024/08/21 09:59:21.828 +00:00] [INFO] [cluster.go:710] ["node state check job has been stopped"]
2024-08-21T09:59:37.1254169Z [2024/08/21 09:59:21.828 +00:00] [INFO] [cluster.go:686] ["metrics are reset"]
2024-08-21T09:59:37.1254945Z [2024/08/21 09:59:21.828 +00:00] [INFO] [server.go:127] ["region syncer has been stopped"]
2024-08-21T09:59:37.1255938Z [2024/08/21 09:59:21.828 +00:00] [INFO] [cluster.go:688] ["metrics collection job has been stopped"]
2024-08-21T09:59:37.1256731Z [2024/08/21 09:59:21.828 +00:00] [INFO] [cluster.go:520] ["sync store config job is stopped"]
2024-08-21T09:59:37.1257462Z [2024/08/21 09:59:21.828 +00:00] [INFO] [cluster.go:407] ["service check job is stopped"]
2024-08-21T09:59:37.1258166Z [2024/08/21 09:59:21.828 +00:00] [INFO] [labeler.go:70] ["region labeler GC stopped"]
2024-08-21T09:59:37.1259073Z [2024/08/21 09:59:21.828 +00:00] [INFO] [runner.go:131] ["stopping async task runner"] [name=heartbeat-async]
2024-08-21T09:59:37.1260105Z [2024/08/21 09:59:21.828 +00:00] [INFO] [runner.go:131] ["stopping async task runner"] [name=log-async]
2024-08-21T09:59:37.1261069Z [2024/08/21 09:59:21.828 +00:00] [INFO] [runner.go:131] ["stopping async task runner"] [name=misc-async]
2024-08-21T09:59:37.1261982Z [2024/08/21 09:59:21.828 +00:00] [INFO] [scheduling_controller.go:161] ["scheduling metrics are reset"]
2024-08-21T09:59:37.1263113Z [2024/08/21 09:59:21.828 +00:00] [INFO] [scheduling_controller.go:163] ["scheduling metrics collection job has been stopped"]
2024-08-21T09:59:37.1263986Z [2024/08/21 09:59:21.828 +00:00] [INFO] [coordinator.go:232] ["coordinator stops running"]
2024-08-21T09:59:37.1264715Z [2024/08/21 09:59:21.829 +00:00] [INFO] [coordinator.go:210] ["coordinator is stopping"]
2024-08-21T09:59:37.1265559Z [2024/08/21 09:59:21.829 +00:00] [INFO] [scheduling_controller.go:139] ["statistics background jobs has been stopped"]
2024-08-21T09:59:37.1266409Z [2024/08/21 09:59:21.829 +00:00] [INFO] [coordinator.go:213] ["coordinator has been stopped"]
2024-08-21T09:59:37.1267216Z [2024/08/21 09:59:21.829 +00:00] [INFO] [scheduling_controller.go:87] ["scheduling service is stopped"]
2024-08-21T09:59:37.1267975Z [2024/08/21 09:59:21.830 +00:00] [INFO] [cluster.go:769] ["raft cluster is stopped"]
2024-08-21T09:59:37.1268676Z [2024/08/21 09:59:21.830 +00:00] [INFO] [tso.go:436] ["reset the timestamp in memory"] []
2024-08-21T09:59:37.1270080Z [2024/08/21 09:59:21.831 +00:00] [INFO] [server.go:1481] ["leadership transfer starting"] [local-member-id=e298409efaeb2293] [current-leader-member-id=e298409efaeb2293] [transferee-member-id=319ee8b43d843ed7]
2024-08-21T09:59:37.1272407Z [2024/08/21 09:59:21.831 +00:00] [INFO] [raft] [zap_raft.go:77] ["e298409efaeb2293 [term 4] starts to transfer leadership to 319ee8b43d843ed7"]
2024-08-21T09:59:37.1275064Z [2024/08/21 09:59:21.831 +00:00] [INFO] [raft] [zap_raft.go:77] ["e298409efaeb2293 sends MsgTimeoutNow to 319ee8b43d843ed7 immediately as 319ee8b43d843ed7 already has up-to-date log"]
2024-08-21T09:59:37.1277696Z [2024/08/21 09:59:21.831 +00:00] [INFO] [raft] [zap_raft.go:77] ["319ee8b43d843ed7 [term 4] received MsgTimeoutNow from e298409efaeb2293 and starts an election to get leadership."]
2024-08-21T09:59:37.1279845Z [2024/08/21 09:59:21.831 +00:00] [INFO] [raft] [zap_raft.go:77] ["319ee8b43d843ed7 is starting a new election at term 4"]
2024-08-21T09:59:37.1281814Z [2024/08/21 09:59:21.831 +00:00] [INFO] [server.go:1690] ["start to campaign PD leader"] [campaign-leader-name=pd2]
  1. start TestTransferLeader
    PD1 -> PD2 ------- 4 And then meet frequently error

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/ci The issue is related to CI.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants