Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Found a new data race #14135

Closed
ngaut opened this issue Dec 19, 2019 · 1 comment
Closed

Found a new data race #14135

ngaut opened this issue Dec 19, 2019 · 1 comment
Labels
type/bug The issue is confirmed as a bug.

Comments

@ngaut
Copy link
Member

ngaut commented Dec 19, 2019 •
edited by kennytm
Loading

Triggered in PR #14130

See more details:

[2019-12-19T03:45:48.358Z] {"level":"warn","ts":"2019-12-19T11:45:47.046+0800","caller":"clientv3/retry_interceptor.go:60","msg":"retrying of unary invoker failed","target":"endpoint://client-1b57f8b9-0bb5-4e29-be17-b3158da9cf3d/localhost:14822315882361213000","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
[2019-12-19T03:45:48.358Z] [2019/12/19 11:45:47.047 +08:00] [INFO] [syncer.go:373] ["[ddl] syncer check all versions failed, continue checking."] [error="context deadline exceeded"]
[2019-12-19T03:45:48.358Z] ==================
[2019-12-19T03:45:48.358Z] WARNING: DATA RACE
[2019-12-19T03:45:48.358Z] Write at 0x00c000bf80d8 by goroutine 140:
[2019-12-19T03:45:48.358Z]   go.etcd.io/etcd/mvcc.(*keyIndex).put()
[2019-12-19T03:45:48.358Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/mvcc/key_index.go:101 +0x1f6
[2019-12-19T03:45:48.358Z]   go.etcd.io/etcd/mvcc.(*treeIndex).Put()
[2019-12-19T03:45:48.358Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/mvcc/index.go:65 +0x1db
[2019-12-19T03:45:48.358Z]   go.etcd.io/etcd/mvcc.(*storeTxnWrite).put()
[2019-12-19T03:45:48.358Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/mvcc/kvstore_txn.go:211 +0x40e
[2019-12-19T03:45:48.358Z]   go.etcd.io/etcd/mvcc.(*storeTxnWrite).Put()
[2019-12-19T03:45:48.358Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/mvcc/kvstore_txn.go:97 +0x8b
[2019-12-19T03:45:48.358Z]   go.etcd.io/etcd/mvcc.(*metricsTxnWrite).Put()
[2019-12-19T03:45:48.358Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/mvcc/metrics_txn.go:46 +0xeb
[2019-12-19T03:45:48.358Z]   go.etcd.io/etcd/mvcc.(*watchableStoreTxnWrite).Put()
[2019-12-19T03:45:48.358Z]       <autogenerated>:1 +0xbb
[2019-12-19T03:45:48.358Z]   go.etcd.io/etcd/etcdserver.(*applierV3backend).Put()
[2019-12-19T03:45:48.358Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/etcdserver/apply.go:217 +0x36c
[2019-12-19T03:45:48.358Z]   go.etcd.io/etcd/etcdserver.(*quotaApplierV3).Put()
[2019-12-19T03:45:48.358Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/etcdserver/apply.go:829 +0xc2
[2019-12-19T03:45:48.358Z]   go.etcd.io/etcd/etcdserver.(*authApplierV3).Put()
[2019-12-19T03:45:48.358Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/etcdserver/apply_auth.go:83 +0x1b5
[2019-12-19T03:45:48.358Z]   go.etcd.io/etcd/etcdserver.(*applierV3backend).Apply()
[2019-12-19T03:45:48.358Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/etcdserver/apply.go:124 +0x38d
[2019-12-19T03:45:48.358Z]   go.etcd.io/etcd/etcdserver.(*quotaApplierV3).Apply()
[2019-12-19T03:45:48.358Z]       <autogenerated>:1 +0x68
[2019-12-19T03:45:48.358Z]   go.etcd.io/etcd/etcdserver.(*authApplierV3).Apply()
[2019-12-19T03:45:48.358Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/etcdserver/apply_auth.go:58 +0x111
[2019-12-19T03:45:48.358Z]   go.etcd.io/etcd/etcdserver.(*EtcdServer).applyEntryNormal()
[2019-12-19T03:45:48.358Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/etcdserver/server.go:2011 +0x5d8
[2019-12-19T03:45:48.358Z]   go.etcd.io/etcd/etcdserver.(*EtcdServer).apply()
[2019-12-19T03:45:48.358Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/etcdserver/server.go:1925 +0x792
[2019-12-19T03:45:48.358Z]   go.etcd.io/etcd/etcdserver.(*EtcdServer).applyEntries()
[2019-12-19T03:45:48.358Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/etcdserver/server.go:1349 +0x177
[2019-12-19T03:45:48.358Z]   go.etcd.io/etcd/etcdserver.(*EtcdServer).applyAll()
[2019-12-19T03:45:48.358Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/etcdserver/server.go:1075 +0x98
[2019-12-19T03:45:48.358Z]   go.etcd.io/etcd/etcdserver.(*EtcdServer).run.func8()
[2019-12-19T03:45:48.358Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/etcdserver/server.go:1020 +0x53
[2019-12-19T03:45:48.358Z]   go.etcd.io/etcd/pkg/schedule.(*fifo).run()
[2019-12-19T03:45:48.358Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/pkg/schedule/schedule.go:157 +0x11b
[2019-12-19T03:45:48.358Z] 
[2019-12-19T03:45:48.358Z] Previous read at 0x00c000bf80d8 by goroutine 23:
[2019-12-19T03:45:48.358Z]   go.etcd.io/etcd/mvcc.(*keyIndex).get()
[2019-12-19T03:45:48.358Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/mvcc/key_index.go:365 +0x684
[2019-12-19T03:45:48.358Z]   go.etcd.io/etcd/mvcc.(*treeIndex).Revisions.func1()
[2019-12-19T03:45:48.358Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/mvcc/index.go:116 +0xab
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/mvcc.(*treeIndex).visit.func1()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/mvcc/index.go:102 +0x8b
[2019-12-19T03:45:48.359Z]   github.com/google/btree.(*node).iterate()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/github.com/google/btree@v0.0.0-20180813153112-4030bb1f1f0c/btree.go:524 +0x85e
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/mvcc.(*treeIndex).visit()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/github.com/google/btree@v0.0.0-20180813153112-4030bb1f1f0c/btree.go:768 +0x40f
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/mvcc.(*treeIndex).Revisions()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/mvcc/index.go:115 +0x20d
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/mvcc.(*storeTxnRead).rangeKeys()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/mvcc/kvstore_txn.go:128 +0x377
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/mvcc.(*storeTxnRead).Range()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/mvcc/kvstore_txn.go:50 +0xe4
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/mvcc.(*txnReadWrite).Range()
[2019-12-19T03:45:48.359Z]       <autogenerated>:1 +0xf4
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/mvcc.(*metricsTxnWrite).Range()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/mvcc/metrics_txn.go:36 +0x12a
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/etcdserver.(*applierV3backend).Range()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/etcdserver/apply.go:275 +0x325
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/etcdserver.(*EtcdServer).Range.func3()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/etcdserver/v3_server.go:104 +0x9d
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/etcdserver.(*EtcdServer).doSerialize()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/etcdserver/v3_server.go:554 +0xad
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/etcdserver.(*EtcdServer).Range()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/etcdserver/v3_server.go:105 +0x1f5
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/etcdserver/api/v3rpc.(*kvServer).Range()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/etcdserver/api/v3rpc/key.go:52 +0xb0
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/etcdserver/api/v3rpc.(*quotaKVServer).Range()
[2019-12-19T03:45:48.359Z]       <autogenerated>:1 +0x87
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/etcdserver/etcdserverpb._KV_Range_Handler.func1()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/etcdserver/etcdserverpb/rpc.pb.go:3473 +0xa1
[2019-12-19T03:45:48.359Z]   github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/chain.go:31 +0x2d3
[2019-12-19T03:45:48.359Z]   github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).UnaryServerInterceptor.func1()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-prometheus@v1.2.0/server_metrics.go:107 +0xff
[2019-12-19T03:45:48.359Z]   github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/chain.go:34 +0x1d9
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/etcdserver/api/v3rpc.newUnaryInterceptor.func1()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/etcdserver/api/v3rpc/interceptor.go:60 +0x10c
[2019-12-19T03:45:48.359Z]   github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/chain.go:34 +0x1d9
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/etcdserver/api/v3rpc.newLogUnaryInterceptor.func1()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/etcdserver/api/v3rpc/interceptor.go:67 +0xda
[2019-12-19T03:45:48.359Z]   github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/chain.go:39 +0x33f
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/etcdserver/etcdserverpb._KV_Range_Handler()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/etcdserver/etcdserverpb/rpc.pb.go:3475 +0x1d8
[2019-12-19T03:45:48.359Z]   google.golang.org/grpc.(*Server).processUnaryRPC()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/google.golang.org/grpc@v1.17.0/server.go:966 +0x92f
[2019-12-19T03:45:48.359Z]   google.golang.org/grpc.(*Server).handleStream()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/google.golang.org/grpc@v1.17.0/server.go:1245 +0x132b
[2019-12-19T03:45:48.359Z]   google.golang.org/grpc.(*Server).serveStreams.func1.1()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/google.golang.org/grpc@v1.17.0/server.go:685 +0xc8
[2019-12-19T03:45:48.359Z] 
[2019-12-19T03:45:48.359Z] Goroutine 140 (running) created at:
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/pkg/schedule.NewFIFOScheduler()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/pkg/schedule/schedule.go:70 +0x2b1
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/etcdserver.(*EtcdServer).run()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/etcdserver/server.go:905 +0x327
[2019-12-19T03:45:48.359Z] 
[2019-12-19T03:45:48.359Z] Goroutine 23 (finished) created at:
[2019-12-19T03:45:48.359Z]   google.golang.org/grpc.(*Server).serveStreams.func1()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/google.golang.org/grpc@v1.17.0/server.go:683 +0xb8
[2019-12-19T03:45:48.359Z]   google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/google.golang.org/grpc@v1.17.0/internal/transport/http2_server.go:419 +0x14e2
[2019-12-19T03:45:48.359Z]   google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/google.golang.org/grpc@v1.17.0/internal/transport/http2_server.go:459 +0x39f
[2019-12-19T03:45:48.359Z]   google.golang.org/grpc.(*Server).serveStreams()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/google.golang.org/grpc@v1.17.0/server.go:681 +0x19a
[2019-12-19T03:45:48.359Z]   google.golang.org/grpc.(*Server).handleRawConn.func1()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/google.golang.org/grpc@v1.17.0/server.go:643 +0x50
[2019-12-19T03:45:48.359Z] ==================
[2019-12-19T03:45:48.359Z] ==================
[2019-12-19T03:45:48.359Z] WARNING: DATA RACE
[2019-12-19T03:45:48.359Z] Write at 0x00c000bf80c0 by goroutine 140:
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/mvcc.(*keyIndex).put()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/mvcc/key_index.go:102 +0x275
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/mvcc.(*treeIndex).Put()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/mvcc/index.go:65 +0x1db
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/mvcc.(*storeTxnWrite).put()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/mvcc/kvstore_txn.go:211 +0x40e
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/mvcc.(*storeTxnWrite).Put()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/mvcc/kvstore_txn.go:97 +0x8b
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/mvcc.(*metricsTxnWrite).Put()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/mvcc/metrics_txn.go:46 +0xeb
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/mvcc.(*watchableStoreTxnWrite).Put()
[2019-12-19T03:45:48.359Z]       <autogenerated>:1 +0xbb
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/etcdserver.(*applierV3backend).Put()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/etcdserver/apply.go:217 +0x36c
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/etcdserver.(*quotaApplierV3).Put()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/etcdserver/apply.go:829 +0xc2
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/etcdserver.(*authApplierV3).Put()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/etcdserver/apply_auth.go:83 +0x1b5
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/etcdserver.(*applierV3backend).Apply()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/etcdserver/apply.go:124 +0x38d
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/etcdserver.(*quotaApplierV3).Apply()
[2019-12-19T03:45:48.359Z]       <autogenerated>:1 +0x68
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/etcdserver.(*authApplierV3).Apply()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/etcdserver/apply_auth.go:58 +0x111
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/etcdserver.(*EtcdServer).applyEntryNormal()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/etcdserver/server.go:2011 +0x5d8
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/etcdserver.(*EtcdServer).apply()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/etcdserver/server.go:1925 +0x792
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/etcdserver.(*EtcdServer).applyEntries()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/etcdserver/server.go:1349 +0x177
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/etcdserver.(*EtcdServer).applyAll()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/etcdserver/server.go:1075 +0x98
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/etcdserver.(*EtcdServer).run.func8()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/etcdserver/server.go:1020 +0x53
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/pkg/schedule.(*fifo).run()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/pkg/schedule/schedule.go:157 +0x11b
[2019-12-19T03:45:48.359Z] 
[2019-12-19T03:45:48.359Z] Previous read at 0x00c000bf80c0 by goroutine 23:
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/mvcc.(*keyIndex).findGeneration()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/mvcc/key_index.go:313 +0x153
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/mvcc.(*keyIndex).get()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/mvcc/key_index.go:160 +0x255
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/mvcc.(*treeIndex).Revisions.func1()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/mvcc/index.go:116 +0xab
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/mvcc.(*treeIndex).visit.func1()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/mvcc/index.go:102 +0x8b
[2019-12-19T03:45:48.359Z]   github.com/google/btree.(*node).iterate()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/github.com/google/btree@v0.0.0-20180813153112-4030bb1f1f0c/btree.go:524 +0x85e
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/mvcc.(*treeIndex).visit()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/github.com/google/btree@v0.0.0-20180813153112-4030bb1f1f0c/btree.go:768 +0x40f
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/mvcc.(*treeIndex).Revisions()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/mvcc/index.go:115 +0x20d
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/mvcc.(*storeTxnRead).rangeKeys()
[2019-12-19T03:45:48.359Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/mvcc/kvstore_txn.go:128 +0x377
[2019-12-19T03:45:48.359Z]   go.etcd.io/etcd/mvcc.(*storeTxnRead).Range()
[2019-12-19T03:45:48.360Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/mvcc/kvstore_txn.go:50 +0xe4
[2019-12-19T03:45:48.360Z]   go.etcd.io/etcd/mvcc.(*txnReadWrite).Range()
[2019-12-19T03:45:48.360Z]       <autogenerated>:1 +0xf4
[2019-12-19T03:45:48.360Z]   go.etcd.io/etcd/mvcc.(*metricsTxnWrite).Range()
[2019-12-19T03:45:48.360Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/mvcc/metrics_txn.go:36 +0x12a
[2019-12-19T03:45:48.360Z]   go.etcd.io/etcd/etcdserver.(*applierV3backend).Range()
[2019-12-19T03:45:48.360Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/etcdserver/apply.go:275 +0x325
[2019-12-19T03:45:48.360Z]   go.etcd.io/etcd/etcdserver.(*EtcdServer).Range.func3()
[2019-12-19T03:45:48.360Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/etcdserver/v3_server.go:104 +0x9d
[2019-12-19T03:45:48.360Z]   go.etcd.io/etcd/etcdserver.(*EtcdServer).doSerialize()
[2019-12-19T03:45:48.360Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/etcdserver/v3_server.go:554 +0xad
[2019-12-19T03:45:48.360Z]   go.etcd.io/etcd/etcdserver.(*EtcdServer).Range()
[2019-12-19T03:45:48.360Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/etcdserver/v3_server.go:105 +0x1f5
[2019-12-19T03:45:48.360Z]   go.etcd.io/etcd/etcdserver/api/v3rpc.(*kvServer).Range()
[2019-12-19T03:45:48.360Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/etcdserver/api/v3rpc/key.go:52 +0xb0
[2019-12-19T03:45:48.360Z]   go.etcd.io/etcd/etcdserver/api/v3rpc.(*quotaKVServer).Range()
[2019-12-19T03:45:48.360Z]       <autogenerated>:1 +0x87
[2019-12-19T03:45:48.360Z]   go.etcd.io/etcd/etcdserver/etcdserverpb._KV_Range_Handler.func1()
[2019-12-19T03:45:48.360Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/etcdserver/etcdserverpb/rpc.pb.go:3473 +0xa1
[2019-12-19T03:45:48.360Z]   github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
[2019-12-19T03:45:48.360Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/chain.go:31 +0x2d3
[2019-12-19T03:45:48.360Z]   github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).UnaryServerInterceptor.func1()
[2019-12-19T03:45:48.360Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-prometheus@v1.2.0/server_metrics.go:107 +0xff
[2019-12-19T03:45:48.360Z]   github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
[2019-12-19T03:45:48.360Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/chain.go:34 +0x1d9
[2019-12-19T03:45:48.360Z]   go.etcd.io/etcd/etcdserver/api/v3rpc.newUnaryInterceptor.func1()
[2019-12-19T03:45:48.360Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/etcdserver/api/v3rpc/interceptor.go:60 +0x10c
[2019-12-19T03:45:48.360Z]   github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
[2019-12-19T03:45:48.360Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/chain.go:34 +0x1d9
[2019-12-19T03:45:48.360Z]   go.etcd.io/etcd/etcdserver/api/v3rpc.newLogUnaryInterceptor.func1()
[2019-12-19T03:45:48.360Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/etcdserver/api/v3rpc/interceptor.go:67 +0xda
[2019-12-19T03:45:48.360Z]   github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1()
[2019-12-19T03:45:48.360Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/chain.go:39 +0x33f
[2019-12-19T03:45:48.360Z]   go.etcd.io/etcd/etcdserver/etcdserverpb._KV_Range_Handler()
[2019-12-19T03:45:48.360Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/etcdserver/etcdserverpb/rpc.pb.go:3475 +0x1d8
[2019-12-19T03:45:48.360Z]   google.golang.org/grpc.(*Server).processUnaryRPC()
[2019-12-19T03:45:48.360Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/google.golang.org/grpc@v1.17.0/server.go:966 +0x92f
[2019-12-19T03:45:48.360Z]   google.golang.org/grpc.(*Server).handleStream()
[2019-12-19T03:45:48.360Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/google.golang.org/grpc@v1.17.0/server.go:1245 +0x132b
[2019-12-19T03:45:48.360Z]   google.golang.org/grpc.(*Server).serveStreams.func1.1()
[2019-12-19T03:45:48.360Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/google.golang.org/grpc@v1.17.0/server.go:685 +0xc8
[2019-12-19T03:45:48.360Z] 
[2019-12-19T03:45:48.360Z] Goroutine 140 (running) created at:
[2019-12-19T03:45:48.360Z]   go.etcd.io/etcd/pkg/schedule.NewFIFOScheduler()
[2019-12-19T03:45:48.360Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/pkg/schedule/schedule.go:70 +0x2b1
[2019-12-19T03:45:48.360Z]   go.etcd.io/etcd/etcdserver.(*EtcdServer).run()
[2019-12-19T03:45:48.360Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190320044326-77d4b742cdbf/etcdserver/server.go:905 +0x327
[2019-12-19T03:45:48.360Z] 
[2019-12-19T03:45:48.360Z] Goroutine 23 (finished) created at:
[2019-12-19T03:45:48.360Z]   google.golang.org/grpc.(*Server).serveStreams.func1()
[2019-12-19T03:45:48.360Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/google.golang.org/grpc@v1.17.0/server.go:683 +0xb8
[2019-12-19T03:45:48.360Z]   google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders()
[2019-12-19T03:45:48.360Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/google.golang.org/grpc@v1.17.0/internal/transport/http2_server.go:419 +0x14e2
[2019-12-19T03:45:48.360Z]   google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams()
[2019-12-19T03:45:48.360Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/google.golang.org/grpc@v1.17.0/internal/transport/http2_server.go:459 +0x39f
[2019-12-19T03:45:48.360Z]   google.golang.org/grpc.(*Server).serveStreams()
[2019-12-19T03:45:48.360Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/google.golang.org/grpc@v1.17.0/server.go:681 +0x19a
[2019-12-19T03:45:48.360Z]   google.golang.org/grpc.(*Server).handleRawConn.func1()
[2019-12-19T03:45:48.360Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/google.golang.org/grpc@v1.17.0/server.go:643 +0x50
@ngaut ngaut added the type/bug The issue is confirmed as a bug. label Dec 19, 2019
@kennytm
Copy link
Contributor

kennytm commented Jan 9, 2020 •
edited
Loading

The data race happened inside etcd, similar to etcd-io/etcd#10700 (comment) is but closed by etcd-io/etcd#10772.

@ngaut ngaut closed this as completed Jan 20, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@kennytm @ngaut