Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Data race when starting server #7419

Closed
rleungx opened this issue Nov 23, 2023 · 0 comments · Fixed by #7420
Closed

Data race when starting server #7419

rleungx opened this issue Nov 23, 2023 · 0 comments · Fixed by #7420
Labels
severity/minor type/bug The issue is confirmed as a bug.

Comments

@rleungx
Copy link
Member

rleungx commented Nov 23, 2023

Bug Report

==================
WARNING: DATA RACE
Read at 0x00c000242620 by goroutine 89542:
  github.com/tikv/pd/pkg/member.(*EmbeddedEtcdMember).IsLeader()
      /home/runner/work/pd/pd/pkg/member/member.go:114 +0x75
  github.com/tikv/pd/pkg/utils/apiutil/serverapi.(*redirector).ServeHTTP()
      /home/runner/work/pd/pd/pkg/utils/apiutil/serverapi/middleware.go:163 +0x1c4
  github.com/urfave/negroni.middleware.ServeHTTP()
      /home/runner/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:38 +0x134
  github.com/urfave/negroni.middleware.ServeHTTP-fm()
      <autogenerated>:1 +0x71
  github.com/tikv/pd/pkg/utils/apiutil/serverapi.(*runtimeServiceValidator).ServeHTTP()
      /home/runner/work/pd/pd/pkg/utils/apiutil/serverapi/middleware.go:44 +0x171
  github.com/urfave/negroni.middleware.ServeHTTP()
      /home/runner/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:38 +0x134
  github.com/urfave/negroni.(*Negroni).ServeHTTP()
      /home/runner/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:96 +0x1b8
  github.com/gorilla/mux.(*Router).ServeHTTP()
      /home/runner/go/pkg/mod/github.com/gorilla/mux@v1.7.4/mux.go:210 +0x361
  github.com/gorilla/mux.(*Router).ServeHTTP()
      /home/runner/go/pkg/mod/github.com/gorilla/mux@v1.7.4/mux.go:210 +0x361
  github.com/urfave/negroni.(*Negroni).UseHandler.Wrap.func1()
      /home/runner/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:46 +0x66
  github.com/urfave/negroni.HandlerFunc.ServeHTTP()
      /home/runner/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:29 +0x51
  github.com/urfave/negroni.middleware.ServeHTTP()
      /home/runner/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:38 +0x134
  github.com/urfave/negroni.middleware.ServeHTTP-fm()
      <autogenerated>:1 +0x71
  github.com/urfave/negroni.(*Recovery).ServeHTTP()
      /home/runner/go/pkg/mod/github.com/urfave/negroni@v0.3.0/recovery.go:193 +0xe1
  github.com/urfave/negroni.middleware.ServeHTTP()
      /home/runner/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:38 +0x134
  github.com/urfave/negroni.(*Negroni).ServeHTTP()
      /home/runner/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:96 +0x1b8
  net/http.(*ServeMux).ServeHTTP()
      /opt/hostedtoolcache/go/1.21.4/x64/src/net/http/server.go:2514 +0xbc
  go.etcd.io/etcd/embed.(*accessController).ServeHTTP()
      /home/runner/go/pkg/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20220915004622-85b640cee793/embed/serve.go:381 +0x94d
  net/http.serverHandler.ServeHTTP()
      /opt/hostedtoolcache/go/1.21.4/x64/src/net/http/server.go:2938 +0x2a1
  net/http.(*conn).serve()
      /opt/hostedtoolcache/go/1.21.4/x64/src/net/http/server.go:2009 +0xc24
  net/http.(*Server).Serve.func3()
      /opt/hostedtoolcache/go/1.21.4/x64/src/net/http/server.go:3086 +0x4f

Previous write at 0x00c000242620 by goroutine 83520:
  github.com/tikv/pd/pkg/member.(*EmbeddedEtcdMember).InitMemberInfo()
      /home/runner/work/pd/pd/pkg/member/member.go:346 +0x9e8
  github.com/tikv/pd/server.(*Server).startServer()
      /home/runner/work/pd/pd/server/server.go:434 +0x939
  github.com/tikv/pd/server.(*Server).Run()
      /home/runner/work/pd/pd/server/server.go:586 +0x1c7
  github.com/tikv/pd/tests.(*TestServer).Run()
      /home/runner/work/pd/pd/tests/cluster.go:125 +0x1e4
  github.com/tikv/pd/tests/integrations/mcs/scheduling.(*serverTestSuite).TestAllocIDAfterLeaderChange()
      /home/runner/work/pd/pd/tests/integrations/mcs/scheduling/server_test.go:101 +0x135
  github.com/pingcap/failpoint.parseTerm()
      /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/terms.go:149 +0x364
  github.com/pingcap/failpoint.parse()
      /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/terms.go:126 +0xa5
  github.com/pingcap/failpoint.newTerms()
      /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/terms.go:98 +0x3e
  github.com/pingcap/failpoint.(*Failpoint).Enable()
      /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/failpoint.go:54 +0x3e
  github.com/pingcap/failpoint.(*Failpoints).Enable()
      /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/failpoints.go:105 +0x276
  github.com/pingcap/failpoint.Enable()
      /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/failpoints.go:225 +0x9c
  github.com/tikv/pd/tests/integrations/mcs/scheduling.(*serverTestSuite).TestAllocIDAfterLeaderChange()
      /home/runner/work/pd/pd/tests/integrations/mcs/scheduling/server_test.go:98 +0x9d
  runtime.call16()
      /opt/hostedtoolcache/go/1.21.4/x64/src/runtime/asm_amd64.s:747 +0x42
  reflect.Value.Call()
      /opt/hostedtoolcache/go/1.21.4/x64/src/reflect/value.go:380 +0xb5
  github.com/stretchr/testify/suite.Run.func1()
      /home/runner/go/pkg/mod/github.com/stretchr/testify@v1.8.2/suite/suite.go:197 +0x684
  testing.tRunner()
      /opt/hostedtoolcache/go/1.21.4/x64/src/testing/testing.go:1595 +0x238
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.21.4/x64/src/testing/testing.go:1648 +0x44

Goroutine 89542 (running) created at:
  net/http.(*Server).Serve()
      /opt/hostedtoolcache/go/1.21.4/x64/src/net/http/server.go:3086 +0x80c
  go.etcd.io/etcd/embed.(*serveCtx).serve.func3()
      /home/runner/go/pkg/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20220915004622-85b640cee793/embed/serve.go:141 +0x4f

Goroutine 83520 (running) created at:
  testing.(*T).Run()
      /opt/hostedtoolcache/go/1.21.4/x64/src/testing/testing.go:1648 +0x82a
  github.com/stretchr/testify/suite.runTests()
      /home/runner/go/pkg/mod/github.com/stretchr/testify@v1.8.2/suite/suite.go:242 +0x186
  github.com/stretchr/testify/suite.Run()
      /home/runner/go/pkg/mod/github.com/stretchr/testify@v1.8.2/suite/suite.go:215 +0x9d2
  github.com/pingcap/failpoint.parse()
      /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/terms.go:126 +0xa5
  github.com/pingcap/failpoint.newTerms()
      /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/terms.go:98 +0x3e
  github.com/pingcap/failpoint.(*Failpoint).Enable()
      /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/failpoint.go:54 +0x3e
  github.com/pingcap/failpoint.(*Failpoints).Enable()
      /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/failpoints.go:105 +0x276
  github.com/pingcap/failpoint.Enable()
      /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/failpoints.go:225 +0x7a
  github.com/tikv/pd/tests/integrations/mcs/scheduling.(*serverTestSuite).SetupSuite()
      /home/runner/work/pd/pd/tests/integrations/mcs/scheduling/server_test.go:62 +0x7b
  github.com/stretchr/testify/suite.Run()
      /home/runner/go/pkg/mod/github.com/stretchr/testify@v1.8.2/suite/suite.go:154 +0x5fe
  github.com/tikv/pd/tests/integrations/mcs/scheduling.TestServerTestSuite()
      /home/runner/work/pd/pd/tests/integrations/mcs/scheduling/server_test.go:56 +0x3c
  testing.tRunner()
      /opt/hostedtoolcache/go/1.21.4/x64/src/testing/testing.go:1595 +0x238
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.21.4/x64/src/testing/testing.go:1648 +0x44
==================
==================
WARNING: DATA RACE
Read at 0x00c0031c6110 by goroutine 89542:
  sync/atomic.LoadInt64()
      /opt/hostedtoolcache/go/1.21.4/x64/src/runtime/race_amd64.s:208 +0xb
  sync/atomic.LoadPointer()
      <autogenerated>:1 +0x10
  github.com/tikv/pd/pkg/election.(*Leadership).getLease()
      /home/runner/work/pd/pd/pkg/election/leadership.go:87 +0xd2
  github.com/tikv/pd/pkg/election.(*Leadership).Check()
      /home/runner/work/pd/pd/pkg/election/leadership.go:176 +0x6a
  github.com/tikv/pd/pkg/member.(*EmbeddedEtcdMember).IsLeader()
      /home/runner/work/pd/pd/pkg/member/member.go:114 +0x84
  github.com/tikv/pd/pkg/utils/apiutil/serverapi.(*redirector).ServeHTTP()
      /home/runner/work/pd/pd/pkg/utils/apiutil/serverapi/middleware.go:163 +0x1c4
  github.com/urfave/negroni.middleware.ServeHTTP()
      /home/runner/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:38 +0x134
  github.com/urfave/negroni.middleware.ServeHTTP-fm()
      <autogenerated>:1 +0x71
  github.com/tikv/pd/pkg/utils/apiutil/serverapi.(*runtimeServiceValidator).ServeHTTP()
      /home/runner/work/pd/pd/pkg/utils/apiutil/serverapi/middleware.go:44 +0x171
  github.com/urfave/negroni.middleware.ServeHTTP()
      /home/runner/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:38 +0x134
  github.com/urfave/negroni.(*Negroni).ServeHTTP()
      /home/runner/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:96 +0x1b8
  github.com/gorilla/mux.(*Router).ServeHTTP()
      /home/runner/go/pkg/mod/github.com/gorilla/mux@v1.7.4/mux.go:210 +0x361
  github.com/gorilla/mux.(*Router).ServeHTTP()
      /home/runner/go/pkg/mod/github.com/gorilla/mux@v1.7.4/mux.go:210 +0x361
  github.com/urfave/negroni.(*Negroni).UseHandler.Wrap.func1()
      /home/runner/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:46 +0x66
  github.com/urfave/negroni.HandlerFunc.ServeHTTP()
      /home/runner/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:29 +0x51
  github.com/urfave/negroni.middleware.ServeHTTP()
      /home/runner/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:38 +0x134
  github.com/urfave/negroni.middleware.ServeHTTP-fm()
      <autogenerated>:1 +0x71
  github.com/urfave/negroni.(*Recovery).ServeHTTP()
      /home/runner/go/pkg/mod/github.com/urfave/negroni@v0.3.0/recovery.go:193 +0xe1
  github.com/urfave/negroni.middleware.ServeHTTP()
      /home/runner/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:38 +0x134
  github.com/urfave/negroni.(*Negroni).ServeHTTP()
      /home/runner/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:96 +0x1b8
  net/http.(*ServeMux).ServeHTTP()
      /opt/hostedtoolcache/go/1.21.4/x64/src/net/http/server.go:2514 +0xbc
  go.etcd.io/etcd/embed.(*accessController).ServeHTTP()
      /home/runner/go/pkg/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20220915004622-85b640cee793/embed/serve.go:381 +0x94d
  net/http.serverHandler.ServeHTTP()
      /opt/hostedtoolcache/go/1.21.4/x64/src/net/http/server.go:2938 +0x2a1
  net/http.(*conn).serve()
      /opt/hostedtoolcache/go/1.21.4/x64/src/net/http/server.go:2009 +0xc24
  net/http.(*Server).Serve.func3()
      /opt/hostedtoolcache/go/1.21.4/x64/src/net/http/server.go:3086 +0x4f

Previous write at 0x00c0031c6110 by goroutine 83520:
  github.com/tikv/pd/pkg/election.NewLeadership()
      /home/runner/work/pd/pd/pkg/election/leadership.go:75 +0x86e
  github.com/tikv/pd/pkg/member.(*EmbeddedEtcdMember).InitMemberInfo()
      /home/runner/work/pd/pd/pkg/member/member.go:346 +0x6f5
  github.com/tikv/pd/server.(*Server).startServer()
      /home/runner/work/pd/pd/server/server.go:434 +0x939
  github.com/tikv/pd/server.(*Server).Run()
      /home/runner/work/pd/pd/server/server.go:586 +0x1c7
  github.com/tikv/pd/tests.(*TestServer).Run()
      /home/runner/work/pd/pd/tests/cluster.go:125 +0x1e4
  github.com/tikv/pd/tests/integrations/mcs/scheduling.(*serverTestSuite).TestAllocIDAfterLeaderChange()
      /home/runner/work/pd/pd/tests/integrations/mcs/scheduling/server_test.go:101 +0x135
  github.com/pingcap/failpoint.parseTerm()
      /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/terms.go:149 +0x364
  github.com/pingcap/failpoint.parse()
      /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/terms.go:126 +0xa5
  github.com/pingcap/failpoint.newTerms()
      /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/terms.go:98 +0x3e
  github.com/pingcap/failpoint.(*Failpoint).Enable()
      /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/failpoint.go:54 +0x3e
  github.com/pingcap/failpoint.(*Failpoints).Enable()
      /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/failpoints.go:105 +0x276
  github.com/pingcap/failpoint.Enable()
      /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/failpoints.go:225 +0x9c
  github.com/tikv/pd/tests/integrations/mcs/scheduling.(*serverTestSuite).TestAllocIDAfterLeaderChange()
      /home/runner/work/pd/pd/tests/integrations/mcs/scheduling/server_test.go:98 +0x9d
  runtime.call16()
      /opt/hostedtoolcache/go/1.21.4/x64/src/runtime/asm_amd64.s:747 +0x42
  reflect.Value.Call()
      /opt/hostedtoolcache/go/1.21.4/x64/src/reflect/value.go:380 +0xb5
  github.com/stretchr/testify/suite.Run.func1()
      /home/runner/go/pkg/mod/github.com/stretchr/testify@v1.8.2/suite/suite.go:197 +0x684
  testing.tRunner()
      /opt/hostedtoolcache/go/1.21.4/x64/src/testing/testing.go:1595 +0x238
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.21.4/x64/src/testing/testing.go:1648 +0x44

Goroutine 89542 (running) created at:
  net/http.(*Server).Serve()
      /opt/hostedtoolcache/go/1.21.4/x64/src/net/http/server.go:3086 +0x80c
  go.etcd.io/etcd/embed.(*serveCtx).serve.func3()
      /home/runner/go/pkg/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20220915004622-85b640cee793/embed/serve.go:141 +0x4f

Goroutine 83520 (running) created at:
  testing.(*T).Run()
      /opt/hostedtoolcache/go/1.21.4/x64/src/testing/testing.go:1648 +0x82a
  github.com/stretchr/testify/suite.runTests()
      /home/runner/go/pkg/mod/github.com/stretchr/testify@v1.8.2/suite/suite.go:242 +0x186
  github.com/stretchr/testify/suite.Run()
      /home/runner/go/pkg/mod/github.com/stretchr/testify@v1.8.2/suite/suite.go:215 +0x9d2
  github.com/pingcap/failpoint.parse()
      /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/terms.go:126 +0xa5
  github.com/pingcap/failpoint.newTerms()
      /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/terms.go:98 +0x3e
  github.com/pingcap/failpoint.(*Failpoint).Enable()
      /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/failpoint.go:54 +0x3e
  github.com/pingcap/failpoint.(*Failpoints).Enable()
      /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/failpoints.go:105 +0x276
  github.com/pingcap/failpoint.Enable()
      /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/failpoints.go:225 +0x7a
  github.com/tikv/pd/tests/integrations/mcs/scheduling.(*serverTestSuite).SetupSuite()
      /home/runner/work/pd/pd/tests/integrations/mcs/scheduling/server_test.go:62 +0x7b
  github.com/stretchr/testify/suite.Run()
      /home/runner/go/pkg/mod/github.com/stretchr/testify@v1.8.2/suite/suite.go:154 +0x5fe
  github.com/tikv/pd/tests/integrations/mcs/scheduling.TestServerTestSuite()
      /home/runner/work/pd/pd/tests/integrations/mcs/scheduling/server_test.go:56 +0x3c
  testing.tRunner()
      /opt/hostedtoolcache/go/1.21.4/x64/src/testing/testing.go:1595 +0x238
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.21.4/x64/src/testing/testing.go:1648 +0x44
==================
@rleungx rleungx added the type/bug The issue is confirmed as a bug. label Nov 23, 2023
@rleungx rleungx mentioned this issue Nov 23, 2023
Merged
ti-chi-bot bot added a commit that referenced this issue Nov 23, 2023
@rleungx @ti-chi-bot
*: fix data race when starting server (#7420)
8198735 
close #7419

Signed-off-by: Ryan Leung <rleungx@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@rleungx rleungx mentioned this issue Nov 23, 2023
Closed
rleungx added a commit to rleungx/pd that referenced this issue Dec 1, 2023
@rleungx @ti-chi-bot
*: fix data race when starting server (tikv#7420)
a088610 
close tikv#7419

Signed-off-by: Ryan Leung <rleungx@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
severity/minor type/bug The issue is confirmed as a bug.
Projects
Questions and Bug Reports
Status: Closed
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

*: fix data race when starting server rleungx/pd
1 participant
@rleungx