TestMixedTSODeployment has random failures due to data race in lease

[binshi-bing]

Flaky Test

Which jobs are failing

TestMixedTSODeployment

WARNING: DATA RACE
Write at 0x00c008ac3838 by goroutine 482457:
github.com/tikv/pd/pkg/election.(*lease).Grant()
/Users/binshi/code/pingcap/my-pd/pkg/election/lease.go:67 +0x4b8
github.com/tikv/pd/pkg/election.(*Leadership).Campaign()
/Users/binshi/code/pingcap/my-pd/pkg/election/leadership.go:111 +0x214
github.com/tikv/pd/pkg/member.(*EmbeddedEtcdMember).CampaignLeader()
/Users/binshi/code/pingcap/my-pd/pkg/member/member.go:168 +0x228
github.com/tikv/pd/server.(*Server).campaignLeader()
/Users/binshi/code/pingcap/my-pd/server/server.go:1451 +0x198
github.com/tikv/pd/server.(*Server).leaderLoop()
/Users/binshi/code/pingcap/my-pd/server/server.go:1445 +0x9c0
github.com/tikv/pd/server.(*Server).startServerLoop.func1()
/Users/binshi/code/pingcap/my-pd/server/server.go:565 +0x34

Previous read at 0x00c008ac3838 by goroutine 488107:
github.com/tikv/pd/pkg/election.(*lease).Close()
/Users/binshi/code/pingcap/my-pd/pkg/election/lease.go:83 +0xc8
github.com/tikv/pd/pkg/election.(*Leadership).Reset()
/Users/binshi/code/pingcap/my-pd/pkg/election/leadership.go:236 +0xd0
github.com/tikv/pd/pkg/member.(*EmbeddedEtcdMember).ResetLeader()
/Users/binshi/code/pingcap/my-pd/pkg/member/member.go:249 +0x38
github.com/tikv/pd/tests.(*TestServer).ResignLeader()
/Users/binshi/code/pingcap/my-pd/tests/cluster.go:155 +0x9c
github.com/tikv/pd/tests/integrations/tso.TestMixedTSODeployment.func1()
/Users/binshi/code/pingcap/my-pd/tests/integrations/tso/client_test.go:270 +0xe8

CI link

local run:
under pd/tests/integrations/tso, run "make test"

Reason for failure (if possible)

data race in lease.go. The log shown above is clear.

Anything else

[rleungx]

Maybe we can skip it since #6304 is still not solved.

[binshi-bing]

Maybe we can skip it since #6304 is still not solved.

#6304 is triggered by this issue, i.e., it encounters Data Race issue at first, then it causes server hung at close. Given this reason, we can fix this issue first to make this test to be stable, then consider fixing #6304 for overall optimization of the server shutdown logic.

[binshi-bing]

We have DataRace issue on accessing keepAliveCancelFunc too.

==================
WARNING: DATA RACE
Read at 0x00c012d8c8f8 by goroutine 1944670:
github.com/tikv/pd/pkg/election.(*Leadership).Reset()
/Users/binshi/code/pingcap/my-pd/pkg/election/leadership.go:233 +0x80
github.com/tikv/pd/pkg/member.(*EmbeddedEtcdMember).ResetLeader()
/Users/binshi/code/pingcap/my-pd/pkg/member/member.go:249 +0x38
github.com/tikv/pd/tests.(*TestServer).ResignLeader()
/Users/binshi/code/pingcap/my-pd/tests/cluster.go:155 +0x9c
github.com/tikv/pd/tests/integrations/tso.TestMixedTSODeployment.func1()
/Users/binshi/code/pingcap/my-pd/tests/integrations/tso/client_test.go:333 +0xe8

Previous write at 0x00c012d8c8f8 by goroutine 1938913:
github.com/tikv/pd/pkg/election.(*Leadership).Keep()
/Users/binshi/code/pingcap/my-pd/pkg/election/leadership.go:140 +0xa8
github.com/tikv/pd/pkg/member.(*EmbeddedEtcdMember).KeepLeader()
/Users/binshi/code/pingcap/my-pd/pkg/member/member.go:173 +0x70c
github.com/tikv/pd/server.(*Server).campaignLeader()
/Users/binshi/code/pingcap/my-pd/server/server.go:1475 +0x6e0
github.com/tikv/pd/server.(*Server).leaderLoop()
/Users/binshi/code/pingcap/my-pd/server/server.go:1445 +0x9c0
github.com/tikv/pd/server.(*Server).startServerLoop.func1()
/Users/binshi/code/pingcap/my-pd/server/server.go:565 +0x34