Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Umbrella issue for flaky tests #10700

Closed
xiang90 opened this issue May 2, 2019 · 26 comments
Closed

Umbrella issue for flaky tests #10700

xiang90 opened this issue May 2, 2019 · 26 comments
Labels
good first issue help wanted

Comments

@xiang90
Copy link
Contributor

xiang90 commented May 2, 2019

The current test CI is very flaky. We need to start to fix them.

I will look over the recent CI failures and post the flaky test cases here.
@jingyih
Copy link
Contributor

jingyih commented May 2, 2019 •
edited
Loading

As mentioned in etcd community meeting, I happen to have already collected the flaky tests. Here they are.

Flaky tests:

  • TestBalancerUnderBlackholeKeepAliveWatch
--- FAIL: TestBalancerUnderBlackholeKeepAliveWatch (5.72s)
    black_hole_test.go:83: took too long to receive watch events
  • TestBalancerUnderNetworkPartitionWatchLeader
--- FAIL: TestBalancerUnderNetworkPartitionWatchLeader (3.27s)
    network_partition_test.go:266: took too long to detect leader lost
  • TestRoundRobinBalancedResolvableFailoverFromRequestFail
--- FAIL: TestRoundRobinBalancedResolvableFailoverFromRequestFail (0.03s)
    balancer_test.go:308: expected balanced loads for 20 requests, got switches 2

=== RUN   TestLeasingTxnCancel
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x60 pc=0xb0184d]

=== RUN   TestLeasingNonOwnerPutError
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x60 pc=0xaf8a8d]

=== RUN   TestLeasingOwnerDeleteError
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x60 pc=0xaf8a8d]

We fixed the following tests. If you see them in CI failure after the fix PR, please let us know.

--- FAIL: TestResumeElection (0.04s)
    testing.go:771: race detected during execution of test

--- FAIL: TestLeasingDeleteRangeContendTxn (0.94s)
    testing.go:771: race detected during execution of test

@jingyih jingyih mentioned this issue May 6, 2019
Merged
@xiang90
Copy link
Contributor Author

xiang90 commented May 14, 2019

another one txn cancel

=== RUN   TestLeasingTxnCancel
{"level":"warn","ts":"2019-05-14T14:00:38.384Z","caller":"clientv3/retry_interceptor.go:60","msg":"retrying of unary invoker failed","target":"endpoint://client-aa069d4d-0556-44fb-b798-344b6508f99f/localhost:715405736098611950","attempt":0,"error":"rpc error: code = Unavailable desc = etcdserver: leader changed"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x60 pc=0xaf8a8d]
goroutine 20054 [running]:
google.golang.org/grpc.(*csAttempt).sendMsg(0xc00025ba20, 0x12b23c0, 0xc00051c2e0, 0xc0019569c8, 0x5, 0x5, 0xc0019569d0, 0xb, 0xb, 0xc0019569d0, ...)
	/go/pkg/mod/google.golang.org/grpc@v1.14.0/stream.go:713 +0x14d
google.golang.org/grpc.(*clientStream).SendMsg.func2(0xc00025ba20, 0x48, 0x48)
	/go/pkg/mod/google.golang.org/grpc@v1.14.0/stream.go:637 +0x186
google.golang.org/grpc.(*clientStream).withRetry(0xc001d26480, 0xc0019cf900, 0xc0012cbaa0, 0xc0019569c8, 0x5)
	/go/pkg/mod/google.golang.org/grpc@v1.14.0/stream.go:530 +0x431
google.golang.org/grpc.(*clientStream).SendMsg(0xc001d26480, 0x12b23c0, 0xc00051c2e0, 0x0, 0x0)
	/go/pkg/mod/google.golang.org/grpc@v1.14.0/stream.go:643 +0x80c
go.etcd.io/etcd/v3/etcdserver/etcdserverpb.(*watchWatchClient).Send(0xc00051c2d0, 0xc00051c2e0, 0x0, 0x1cd5a18)
	/go/src/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3618 +0x6b
go.etcd.io/etcd/v3/clientv3.(*watchGrpcStream).run(0xc001a496c0)
	/go/src/go.etcd.io/etcd/clientv3/watch.go:546 +0xd8d
created by go.etcd.io/etcd/v3/clientv3.(*watcher).newWatcherGrpcStream
	/go/src/go.etcd.io/etcd/clientv3/watch.go:277 +0x58c
FAIL	go.etcd.io/etcd/v3/clientv3/integration	128.500s

@xiang90
Copy link
Contributor Author

xiang90 commented May 14, 2019

semaphoreci sometimes ran out of the 20m limit.

@xiang90
Copy link
Contributor Author

xiang90 commented May 14, 2019

another TestBalancerUnderBlackholeKeepAliveWatch

=== RUN   TestBalancerUnderBlackholeKeepAliveWatch
--- FAIL: TestBalancerUnderBlackholeKeepAliveWatch (5.76s)
    black_hole_test.go:83: took too long to receive watch events

@xiang90
Copy link
Contributor Author

xiang90 commented May 14, 2019 •
edited
Loading

TestCtlV3AuthFromKeyPerm

--- FAIL: TestCtlV3AuthFromKeyPerm (16.21s)
    testutil.go:55: goroutine 3497 [running]:
        go.etcd.io/etcd/pkg/testutil.FatalStack(0xacc2140, 0xae3c020, 0x18)
        	/go/src/go.etcd.io/etcd/pkg/testutil/testutil.go:54 +0x60
        go.etcd.io/etcd/tests/e2e.testCtl(0xacc2140, 0x87592e4, 0x0, 0x0, 0x0)
        	/go/src/go.etcd.io/etcd/tests/e2e/ctl_v3_test.go:184 +0x3c1
        go.etcd.io/etcd/tests/e2e.TestCtlV3AuthFromKeyPerm(0xacc2140)
        	/go/src/go.etcd.io/etcd/tests/e2e/ctl_v3_auth_test.go:43 +0x43
        testing.tRunner(0xacc2140, 0x8758f7c)
        	/usr/local/go/src/testing/testing.go:827 +0xa3
        created by testing.(*T).Run

https://semaphoreci.com/etcd-io/etcd/branches/pull-request-10683/builds/5

@xiang90
Copy link
Contributor Author

xiang90 commented May 14, 2019

TestBarrierSingleNode

WARNING: DATA RACE
Write at 0x00c006787ba8 by goroutine 334:
  go.etcd.io/etcd/mvcc.(*keyIndex).put()
      /go/src/go.etcd.io/etcd/mvcc/key_index.go:101 +0x206
  go.etcd.io/etcd/mvcc.(*keyIndex).tombstone()
      /go/src/go.etcd.io/etcd/mvcc/key_index.go:141 +0x2ea
  go.etcd.io/etcd/mvcc.(*treeIndex).Tombstone()
      /go/src/go.etcd.io/etcd/mvcc/index.go:151 +0x19c
  go.etcd.io/etcd/mvcc.(*storeTxnWrite).delete()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:283 +0x375
  go.etcd.io/etcd/mvcc.(*storeTxnWrite).deleteRange()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:251 +0x1b9
  go.etcd.io/etcd/mvcc.(*storeTxnWrite).DeleteRange()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:90 +0x81
  go.etcd.io/etcd/mvcc.(*metricsTxnWrite).DeleteRange()
      /go/src/go.etcd.io/etcd/mvcc/metrics_txn.go:41 +0xe1
  go.etcd.io/etcd/mvcc.(*watchableStoreTxnWrite).DeleteRange()
      <autogenerated>:1 +0xb1
  go.etcd.io/etcd/etcdserver.(*applierV3backend).DeleteRange()
      /go/src/go.etcd.io/etcd/etcdserver/apply.go:244 +0x241
  go.etcd.io/etcd/etcdserver.(*quotaApplierV3).DeleteRange()
      <autogenerated>:1 +0x87
  go.etcd.io/etcd/etcdserver.(*authApplierV3).DeleteRange()
      /go/src/go.etcd.io/etcd/etcdserver/apply_auth.go:104 +0x1ce
  go.etcd.io/etcd/etcdserver.(*applierV3backend).Apply()
      /go/src/go.etcd.io/etcd/etcdserver/apply.go:126 +0x4dd
  go.etcd.io/etcd/etcdserver.(*quotaApplierV3).Apply()
      <autogenerated>:1 +0x68
  go.etcd.io/etcd/etcdserver.(*authApplierV3).Apply()
      /go/src/go.etcd.io/etcd/etcdserver/apply_auth.go:58 +0xf3
  go.etcd.io/etcd/etcdserver.(*EtcdServer).applyEntryNormal()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:2011 +0x3c8
  go.etcd.io/etcd/etcdserver.(*EtcdServer).apply()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:1925 +0x77e
  go.etcd.io/etcd/etcdserver.(*EtcdServer).applyEntries()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:1349 +0x175
  go.etcd.io/etcd/etcdserver.(*EtcdServer).applyAll()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:1075 +0x98
  go.etcd.io/etcd/etcdserver.(*EtcdServer).run.func8()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:1020 +0x53
  go.etcd.io/etcd/pkg/schedule.(*fifo).run()
      /go/src/go.etcd.io/etcd/pkg/schedule/schedule.go:157 +0x105
Previous read at 0x00c006787ba8 by goroutine 415:
  go.etcd.io/etcd/mvcc.(*keyIndex).get()
      /go/src/go.etcd.io/etcd/mvcc/key_index.go:365 +0x6ae
  go.etcd.io/etcd/mvcc.(*treeIndex).Revisions.func1()
      /go/src/go.etcd.io/etcd/mvcc/index.go:116 +0xab
  go.etcd.io/etcd/mvcc.(*treeIndex).visit.func1()
      /go/src/go.etcd.io/etcd/mvcc/index.go:102 +0x85
  go.etcd.io/etcd/vendor/github.com/google/btree.(*node).iterate()
      /go/src/go.etcd.io/etcd/vendor/github.com/google/btree/btree.go:523 +0x17f
  go.etcd.io/etcd/vendor/github.com/google/btree.(*BTree).AscendGreaterOrEqual()
      /go/src/go.etcd.io/etcd/vendor/github.com/google/btree/btree.go:759 +0xa9
  go.etcd.io/etcd/mvcc.(*treeIndex).visit()
      /go/src/go.etcd.io/etcd/mvcc/index.go:98 +0x3d9
  go.etcd.io/etcd/mvcc.(*treeIndex).Revisions()
      /go/src/go.etcd.io/etcd/mvcc/index.go:115 +0x20d
  go.etcd.io/etcd/mvcc.(*storeTxnRead).rangeKeys()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:128 +0x37a
  go.etcd.io/etcd/mvcc.(*storeTxnRead).Range()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:50 +0xe4
  go.etcd.io/etcd/mvcc.(*txnReadWrite).Range()
      <autogenerated>:1 +0xf4
  go.etcd.io/etcd/mvcc.(*metricsTxnWrite).Range()
      /go/src/go.etcd.io/etcd/mvcc/metrics_txn.go:36 +0x12a
  go.etcd.io/etcd/etcdserver.(*applierV3backend).Range()
      /go/src/go.etcd.io/etcd/etcdserver/apply.go:275 +0x2fc
  go.etcd.io/etcd/etcdserver.(*EtcdServer).Range.func3()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:104 +0x9d
  go.etcd.io/etcd/etcdserver.(*EtcdServer).doSerialize()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:554 +0xad
  go.etcd.io/etcd/etcdserver.(*EtcdServer).Range()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:105 +0x225
  go.etcd.io/etcd/etcdserver/api/v3rpc.(*kvServer).Range()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/key.go:52 +0xb0
  go.etcd.io/etcd/etcdserver/api/v3rpc.(*quotaKVServer).Range()
      <autogenerated>:1 +0x87
  go.etcd.io/etcd/etcdserver/etcdserverpb._KV_Range_Handler.func1()
      /go/src/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3473 +0xa1
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:31 +0x2d3
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).UnaryServerInterceptor.func1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server_metrics.go:107 +0xff
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34 +0x1d9
  go.etcd.io/etcd/etcdserver/api/v3rpc.newUnaryInterceptor.func1()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/interceptor.go:60 +0xf0
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34 +0x1d9
  go.etcd.io/etcd/etcdserver/api/v3rpc.newLogUnaryInterceptor.func1()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/interceptor.go:67 +0xda
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:39 +0x33d
  go.etcd.io/etcd/etcdserver/etcdserverpb._KV_Range_Handler()
      /go/src/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3475 +0x1d8
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).processUnaryRPC()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:1026 +0xa8b
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleStream()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:1252 +0x138d
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:699 +0xac
Goroutine 334 (running) created at:
  go.etcd.io/etcd/pkg/schedule.NewFIFOScheduler()
      /go/src/go.etcd.io/etcd/pkg/schedule/schedule.go:70 +0x2b1
  go.etcd.io/etcd/etcdserver.(*EtcdServer).run()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:905 +0x33f
Goroutine 415 (finished) created at:
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:697 +0xb8
  go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport/http2_server.go:432 +0x14cb
  go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport/http2_server.go:472 +0x37a
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:695 +0x170
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleRawConn.func2()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:649 +0x53
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleRawConn.func3()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:657 +0x34
==================
==================
WARNING: DATA RACE
Write at 0x00c006787b90 by goroutine 334:
  go.etcd.io/etcd/mvcc.(*keyIndex).put()
      /go/src/go.etcd.io/etcd/mvcc/key_index.go:102 +0x27f
  go.etcd.io/etcd/mvcc.(*keyIndex).tombstone()
      /go/src/go.etcd.io/etcd/mvcc/key_index.go:141 +0x2ea
  go.etcd.io/etcd/mvcc.(*treeIndex).Tombstone()
      /go/src/go.etcd.io/etcd/mvcc/index.go:151 +0x19c
  go.etcd.io/etcd/mvcc.(*storeTxnWrite).delete()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:283 +0x375
  go.etcd.io/etcd/mvcc.(*storeTxnWrite).deleteRange()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:251 +0x1b9
  go.etcd.io/etcd/mvcc.(*storeTxnWrite).DeleteRange()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:90 +0x81
  go.etcd.io/etcd/mvcc.(*metricsTxnWrite).DeleteRange()
      /go/src/go.etcd.io/etcd/mvcc/metrics_txn.go:41 +0xe1
  go.etcd.io/etcd/mvcc.(*watchableStoreTxnWrite).DeleteRange()
      <autogenerated>:1 +0xb1
  go.etcd.io/etcd/etcdserver.(*applierV3backend).DeleteRange()
      /go/src/go.etcd.io/etcd/etcdserver/apply.go:244 +0x241
  go.etcd.io/etcd/etcdserver.(*quotaApplierV3).DeleteRange()
      <autogenerated>:1 +0x87
  go.etcd.io/etcd/etcdserver.(*authApplierV3).DeleteRange()
      /go/src/go.etcd.io/etcd/etcdserver/apply_auth.go:104 +0x1ce
  go.etcd.io/etcd/etcdserver.(*applierV3backend).Apply()
      /go/src/go.etcd.io/etcd/etcdserver/apply.go:126 +0x4dd
  go.etcd.io/etcd/etcdserver.(*quotaApplierV3).Apply()
      <autogenerated>:1 +0x68
  go.etcd.io/etcd/etcdserver.(*authApplierV3).Apply()
      /go/src/go.etcd.io/etcd/etcdserver/apply_auth.go:58 +0xf3
  go.etcd.io/etcd/etcdserver.(*EtcdServer).applyEntryNormal()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:2011 +0x3c8
  go.etcd.io/etcd/etcdserver.(*EtcdServer).apply()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:1925 +0x77e
  go.etcd.io/etcd/etcdserver.(*EtcdServer).applyEntries()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:1349 +0x175
  go.etcd.io/etcd/etcdserver.(*EtcdServer).applyAll()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:1075 +0x98
  go.etcd.io/etcd/etcdserver.(*EtcdServer).run.func8()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:1020 +0x53
  go.etcd.io/etcd/pkg/schedule.(*fifo).run()
      /go/src/go.etcd.io/etcd/pkg/schedule/schedule.go:157 +0x105
Previous read at 0x00c006787b90 by goroutine 415:
  go.etcd.io/etcd/mvcc.(*keyIndex).findGeneration()
      /go/src/go.etcd.io/etcd/mvcc/key_index.go:313 +0x150
  go.etcd.io/etcd/mvcc.(*keyIndex).get()
      /go/src/go.etcd.io/etcd/mvcc/key_index.go:160 +0x255
  go.etcd.io/etcd/mvcc.(*treeIndex).Revisions.func1()
      /go/src/go.etcd.io/etcd/mvcc/index.go:116 +0xab
  go.etcd.io/etcd/mvcc.(*treeIndex).visit.func1()
      /go/src/go.etcd.io/etcd/mvcc/index.go:102 +0x85
  go.etcd.io/etcd/vendor/github.com/google/btree.(*node).iterate()
      /go/src/go.etcd.io/etcd/vendor/github.com/google/btree/btree.go:523 +0x17f
  go.etcd.io/etcd/vendor/github.com/google/btree.(*BTree).AscendGreaterOrEqual()
      /go/src/go.etcd.io/etcd/vendor/github.com/google/btree/btree.go:759 +0xa9
  go.etcd.io/etcd/mvcc.(*treeIndex).visit()
      /go/src/go.etcd.io/etcd/mvcc/index.go:98 +0x3d9
  go.etcd.io/etcd/mvcc.(*treeIndex).Revisions()
      /go/src/go.etcd.io/etcd/mvcc/index.go:115 +0x20d
  go.etcd.io/etcd/mvcc.(*storeTxnRead).rangeKeys()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:128 +0x37a
  go.etcd.io/etcd/mvcc.(*storeTxnRead).Range()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:50 +0xe4
  go.etcd.io/etcd/mvcc.(*txnReadWrite).Range()
      <autogenerated>:1 +0xf4
  go.etcd.io/etcd/mvcc.(*metricsTxnWrite).Range()
      /go/src/go.etcd.io/etcd/mvcc/metrics_txn.go:36 +0x12a
  go.etcd.io/etcd/etcdserver.(*applierV3backend).Range()
      /go/src/go.etcd.io/etcd/etcdserver/apply.go:275 +0x2fc
  go.etcd.io/etcd/etcdserver.(*EtcdServer).Range.func3()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:104 +0x9d
  go.etcd.io/etcd/etcdserver.(*EtcdServer).doSerialize()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:554 +0xad
  go.etcd.io/etcd/etcdserver.(*EtcdServer).Range()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:105 +0x225
  go.etcd.io/etcd/etcdserver/api/v3rpc.(*kvServer).Range()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/key.go:52 +0xb0
  go.etcd.io/etcd/etcdserver/api/v3rpc.(*quotaKVServer).Range()
      <autogenerated>:1 +0x87
  go.etcd.io/etcd/etcdserver/etcdserverpb._KV_Range_Handler.func1()
      /go/src/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3473 +0xa1
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:31 +0x2d3
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).UnaryServerInterceptor.func1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server_metrics.go:107 +0xff
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34 +0x1d9
  go.etcd.io/etcd/etcdserver/api/v3rpc.newUnaryInterceptor.func1()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/interceptor.go:60 +0xf0
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34 +0x1d9
  go.etcd.io/etcd/etcdserver/api/v3rpc.newLogUnaryInterceptor.func1()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/interceptor.go:67 +0xda
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:39 +0x33d
  go.etcd.io/etcd/etcdserver/etcdserverpb._KV_Range_Handler()
      /go/src/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3475 +0x1d8
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).processUnaryRPC()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:1026 +0xa8b
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleStream()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:1252 +0x138d
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:699 +0xac
Goroutine 334 (running) created at:
  go.etcd.io/etcd/pkg/schedule.NewFIFOScheduler()
      /go/src/go.etcd.io/etcd/pkg/schedule/schedule.go:70 +0x2b1
  go.etcd.io/etcd/etcdserver.(*EtcdServer).run()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:905 +0x33f
Goroutine 415 (finished) created at:
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:697 +0xb8
  go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport/http2_server.go:432 +0x14cb
  go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport/http2_server.go:472 +0x37a
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:695 +0x170
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleRawConn.func2()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:649 +0x53
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleRawConn.func3()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:657 +0x34
==================
==================
WARNING: DATA RACE
Write at 0x00c0031fe468 by goroutine 334:
  go.etcd.io/etcd/mvcc.(*keyIndex).tombstone()
      /go/src/go.etcd.io/etcd/mvcc/key_index.go:142 +0x3b7
  go.etcd.io/etcd/mvcc.(*treeIndex).Tombstone()
      /go/src/go.etcd.io/etcd/mvcc/index.go:151 +0x19c
  go.etcd.io/etcd/mvcc.(*storeTxnWrite).delete()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:283 +0x375
  go.etcd.io/etcd/mvcc.(*storeTxnWrite).deleteRange()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:251 +0x1b9
  go.etcd.io/etcd/mvcc.(*storeTxnWrite).DeleteRange()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:90 +0x81
  go.etcd.io/etcd/mvcc.(*metricsTxnWrite).DeleteRange()
      /go/src/go.etcd.io/etcd/mvcc/metrics_txn.go:41 +0xe1
  go.etcd.io/etcd/mvcc.(*watchableStoreTxnWrite).DeleteRange()
      <autogenerated>:1 +0xb1
  go.etcd.io/etcd/etcdserver.(*applierV3backend).DeleteRange()
      /go/src/go.etcd.io/etcd/etcdserver/apply.go:244 +0x241
  go.etcd.io/etcd/etcdserver.(*quotaApplierV3).DeleteRange()
      <autogenerated>:1 +0x87
  go.etcd.io/etcd/etcdserver.(*authApplierV3).DeleteRange()
      /go/src/go.etcd.io/etcd/etcdserver/apply_auth.go:104 +0x1ce
  go.etcd.io/etcd/etcdserver.(*applierV3backend).Apply()
      /go/src/go.etcd.io/etcd/etcdserver/apply.go:126 +0x4dd
  go.etcd.io/etcd/etcdserver.(*quotaApplierV3).Apply()
      <autogenerated>:1 +0x68
  go.etcd.io/etcd/etcdserver.(*authApplierV3).Apply()
      /go/src/go.etcd.io/etcd/etcdserver/apply_auth.go:58 +0xf3
  go.etcd.io/etcd/etcdserver.(*EtcdServer).applyEntryNormal()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:2011 +0x3c8
  go.etcd.io/etcd/etcdserver.(*EtcdServer).apply()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:1925 +0x77e
  go.etcd.io/etcd/etcdserver.(*EtcdServer).applyEntries()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:1349 +0x175
  go.etcd.io/etcd/etcdserver.(*EtcdServer).applyAll()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:1075 +0x98
  go.etcd.io/etcd/etcdserver.(*EtcdServer).run.func8()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:1020 +0x53
  go.etcd.io/etcd/pkg/schedule.(*fifo).run()
      /go/src/go.etcd.io/etcd/pkg/schedule/schedule.go:157 +0x105
Previous read at 0x00c0031fe468 by goroutine 415:
  go.etcd.io/etcd/mvcc.(*keyIndex).get()
      /go/src/go.etcd.io/etcd/mvcc/key_index.go:298 +0x61
  go.etcd.io/etcd/mvcc.(*treeIndex).Revisions.func1()
      /go/src/go.etcd.io/etcd/mvcc/index.go:116 +0xab
  go.etcd.io/etcd/mvcc.(*treeIndex).visit.func1()
      /go/src/go.etcd.io/etcd/mvcc/index.go:102 +0x85
  go.etcd.io/etcd/vendor/github.com/google/btree.(*node).iterate()
      /go/src/go.etcd.io/etcd/vendor/github.com/google/btree/btree.go:523 +0x17f
  go.etcd.io/etcd/vendor/github.com/google/btree.(*BTree).AscendGreaterOrEqual()
      /go/src/go.etcd.io/etcd/vendor/github.com/google/btree/btree.go:759 +0xa9
  go.etcd.io/etcd/mvcc.(*treeIndex).visit()
      /go/src/go.etcd.io/etcd/mvcc/index.go:98 +0x3d9
  go.etcd.io/etcd/mvcc.(*treeIndex).Revisions()
      /go/src/go.etcd.io/etcd/mvcc/index.go:115 +0x20d
  go.etcd.io/etcd/mvcc.(*storeTxnRead).rangeKeys()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:128 +0x37a
  go.etcd.io/etcd/mvcc.(*storeTxnRead).Range()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:50 +0xe4
  go.etcd.io/etcd/mvcc.(*txnReadWrite).Range()
      <autogenerated>:1 +0xf4
  go.etcd.io/etcd/mvcc.(*metricsTxnWrite).Range()
      /go/src/go.etcd.io/etcd/mvcc/metrics_txn.go:36 +0x12a
  go.etcd.io/etcd/etcdserver.(*applierV3backend).Range()
      /go/src/go.etcd.io/etcd/etcdserver/apply.go:275 +0x2fc
  go.etcd.io/etcd/etcdserver.(*EtcdServer).Range.func3()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:104 +0x9d
  go.etcd.io/etcd/etcdserver.(*EtcdServer).doSerialize()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:554 +0xad
  go.etcd.io/etcd/etcdserver.(*EtcdServer).Range()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:105 +0x225
  go.etcd.io/etcd/etcdserver/api/v3rpc.(*kvServer).Range()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/key.go:52 +0xb0
  go.etcd.io/etcd/etcdserver/api/v3rpc.(*quotaKVServer).Range()
      <autogenerated>:1 +0x87
  go.etcd.io/etcd/etcdserver/etcdserverpb._KV_Range_Handler.func1()
      /go/src/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3473 +0xa1
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:31 +0x2d3
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).UnaryServerInterceptor.func1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server_metrics.go:107 +0xff
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34 +0x1d9
  go.etcd.io/etcd/etcdserver/api/v3rpc.newUnaryInterceptor.func1()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/interceptor.go:60 +0xf0
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34 +0x1d9
  go.etcd.io/etcd/etcdserver/api/v3rpc.newLogUnaryInterceptor.func1()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/interceptor.go:67 +0xda
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:39 +0x33d
  go.etcd.io/etcd/etcdserver/etcdserverpb._KV_Range_Handler()
      /go/src/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3475 +0x1d8
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).processUnaryRPC()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:1026 +0xa8b
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleStream()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:1252 +0x138d
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:699 +0xac
Goroutine 334 (running) created at:
  go.etcd.io/etcd/pkg/schedule.NewFIFOScheduler()
      /go/src/go.etcd.io/etcd/pkg/schedule/schedule.go:70 +0x2b1
  go.etcd.io/etcd/etcdserver.(*EtcdServer).run()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:905 +0x33f
Goroutine 415 (finished) created at:
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:697 +0xb8
  go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport/http2_server.go:432 +0x14cb
  go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport/http2_server.go:472 +0x37a
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:695 +0x170
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleRawConn.func2()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:649 +0x53
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleRawConn.func3()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:657 +0x34
==================
==================
WARNING: DATA RACE
Read at 0x00c00063bfd8 by goroutine 1012:
  go.etcd.io/etcd/mvcc.(*treeIndex).visit()
      /go/src/go.etcd.io/etcd/vendor/github.com/google/btree/btree.go:618 +0x225
  go.etcd.io/etcd/mvcc.(*treeIndex).Revisions()
      /go/src/go.etcd.io/etcd/mvcc/index.go:115 +0x20d
  go.etcd.io/etcd/mvcc.(*storeTxnRead).rangeKeys()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:128 +0x37a
  go.etcd.io/etcd/mvcc.(*storeTxnRead).Range()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:50 +0xe4
  go.etcd.io/etcd/mvcc.(*txnReadWrite).Range()
      <autogenerated>:1 +0xf4
  go.etcd.io/etcd/mvcc.(*metricsTxnWrite).Range()
      /go/src/go.etcd.io/etcd/mvcc/metrics_txn.go:36 +0x12a
  go.etcd.io/etcd/etcdserver.(*applierV3backend).Range()
      /go/src/go.etcd.io/etcd/etcdserver/apply.go:275 +0x2fc
  go.etcd.io/etcd/etcdserver.(*EtcdServer).Range.func3()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:104 +0x9d
  go.etcd.io/etcd/etcdserver.(*EtcdServer).doSerialize()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:554 +0xad
  go.etcd.io/etcd/etcdserver.(*EtcdServer).Range()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:105 +0x225
  go.etcd.io/etcd/etcdserver/api/v3rpc.(*kvServer).Range()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/key.go:52 +0xb0
  go.etcd.io/etcd/etcdserver/api/v3rpc.(*quotaKVServer).Range()
      <autogenerated>:1 +0x87
  go.etcd.io/etcd/etcdserver/etcdserverpb._KV_Range_Handler.func1()
      /go/src/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3473 +0xa1
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:31 +0x2d3
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).UnaryServerInterceptor.func1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server_metrics.go:107 +0xff
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34 +0x1d9
  go.etcd.io/etcd/etcdserver/api/v3rpc.newUnaryInterceptor.func1()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/interceptor.go:60 +0xf0
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34 +0x1d9
  go.etcd.io/etcd/etcdserver/api/v3rpc.newLogUnaryInterceptor.func1()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/interceptor.go:67 +0xda
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:39 +0x33d
  go.etcd.io/etcd/etcdserver/etcdserverpb._KV_Range_Handler()
      /go/src/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3475 +0x1d8
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).processUnaryRPC()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:1026 +0xa8b
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleStream()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:1252 +0x138d
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:699 +0xac
Previous write at 0x00c00063bfd8 by goroutine 843:
  go.etcd.io/etcd/mvcc.(*treeIndex).visit()
      /go/src/go.etcd.io/etcd/vendor/github.com/google/btree/btree.go:620 +0x33b
  go.etcd.io/etcd/mvcc.(*treeIndex).Revisions()
      /go/src/go.etcd.io/etcd/mvcc/index.go:115 +0x20d
  go.etcd.io/etcd/mvcc.(*storeTxnRead).rangeKeys()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:128 +0x37a
  go.etcd.io/etcd/mvcc.(*storeTxnRead).Range()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:50 +0xe4
  go.etcd.io/etcd/mvcc.(*txnReadWrite).Range()
      <autogenerated>:1 +0xf4
  go.etcd.io/etcd/mvcc.(*metricsTxnWrite).Range()
      /go/src/go.etcd.io/etcd/mvcc/metrics_txn.go:36 +0x12a
  go.etcd.io/etcd/etcdserver.(*applierV3backend).Range()
      /go/src/go.etcd.io/etcd/etcdserver/apply.go:275 +0x2fc
  go.etcd.io/etcd/etcdserver.(*EtcdServer).Range.func3()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:104 +0x9d
  go.etcd.io/etcd/etcdserver.(*EtcdServer).doSerialize()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:554 +0xad
  go.etcd.io/etcd/etcdserver.(*EtcdServer).Range()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:105 +0x225
  go.etcd.io/etcd/etcdserver/api/v3rpc.(*kvServer).Range()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/key.go:52 +0xb0
  go.etcd.io/etcd/etcdserver/api/v3rpc.(*quotaKVServer).Range()
      <autogenerated>:1 +0x87
  go.etcd.io/etcd/etcdserver/etcdserverpb._KV_Range_Handler.func1()
      /go/src/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3473 +0xa1
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:31 +0x2d3
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).UnaryServerInterceptor.func1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server_metrics.go:107 +0xff
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34 +0x1d9
  go.etcd.io/etcd/etcdserver/api/v3rpc.newUnaryInterceptor.func1()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/interceptor.go:60 +0xf0
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34 +0x1d9
  go.etcd.io/etcd/etcdserver/api/v3rpc.newLogUnaryInterceptor.func1()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/interceptor.go:67 +0xda
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:39 +0x33d
  go.etcd.io/etcd/etcdserver/etcdserverpb._KV_Range_Handler()
      /go/src/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3475 +0x1d8
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).processUnaryRPC()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:1026 +0xa8b
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleStream()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:1252 +0x138d
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:699 +0xac
Goroutine 1012 (running) created at:
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:697 +0xb8
  go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport/http2_server.go:432 +0x14cb
  go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport/http2_server.go:472 +0x37a
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:695 +0x170
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleRawConn.func2()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:649 +0x53
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleRawConn.func3()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:657 +0x34
Goroutine 843 (finished) created at:
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:697 +0xb8
  go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport/http2_server.go:432 +0x14cb
  go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport/http2_server.go:472 +0x37a
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:695 +0x170
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleRawConn.func2()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:649 +0x53
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleRawConn.func3()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:657 +0x34
==================
==================
WARNING: DATA RACE
Read at 0x00c014205820 by goroutine 1012:
  go.etcd.io/etcd/mvcc.(*treeIndex).visit()
      /go/src/go.etcd.io/etcd/vendor/github.com/google/btree/btree.go:618 +0x23e
  go.etcd.io/etcd/mvcc.(*treeIndex).Revisions()
      /go/src/go.etcd.io/etcd/mvcc/index.go:115 +0x20d
  go.etcd.io/etcd/mvcc.(*storeTxnRead).rangeKeys()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:128 +0x37a
  go.etcd.io/etcd/mvcc.(*storeTxnRead).Range()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:50 +0xe4
  go.etcd.io/etcd/mvcc.(*txnReadWrite).Range()
      <autogenerated>:1 +0xf4
  go.etcd.io/etcd/mvcc.(*metricsTxnWrite).Range()
      /go/src/go.etcd.io/etcd/mvcc/metrics_txn.go:36 +0x12a
  go.etcd.io/etcd/etcdserver.(*applierV3backend).Range()
      /go/src/go.etcd.io/etcd/etcdserver/apply.go:275 +0x2fc
  go.etcd.io/etcd/etcdserver.(*EtcdServer).Range.func3()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:104 +0x9d
  go.etcd.io/etcd/etcdserver.(*EtcdServer).doSerialize()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:554 +0xad
  go.etcd.io/etcd/etcdserver.(*EtcdServer).Range()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:105 +0x225
  go.etcd.io/etcd/etcdserver/api/v3rpc.(*kvServer).Range()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/key.go:52 +0xb0
  go.etcd.io/etcd/etcdserver/api/v3rpc.(*quotaKVServer).Range()
      <autogenerated>:1 +0x87
  go.etcd.io/etcd/etcdserver/etcdserverpb._KV_Range_Handler.func1()
      /go/src/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3473 +0xa1
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:31 +0x2d3
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).UnaryServerInterceptor.func1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server_metrics.go:107 +0xff
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34 +0x1d9
  go.etcd.io/etcd/etcdserver/api/v3rpc.newUnaryInterceptor.func1()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/interceptor.go:60 +0xf0
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34 +0x1d9
  go.etcd.io/etcd/etcdserver/api/v3rpc.newLogUnaryInterceptor.func1()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/interceptor.go:67 +0xda
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:39 +0x33d
  go.etcd.io/etcd/etcdserver/etcdserverpb._KV_Range_Handler()
      /go/src/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3475 +0x1d8
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).processUnaryRPC()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:1026 +0xa8b
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleStream()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:1252 +0x138d
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:699 +0xac
Previous write at 0x00c014205820 by goroutine 843:
  go.etcd.io/etcd/mvcc.(*treeIndex).visit()
      /go/src/go.etcd.io/etcd/mvcc/index.go:95 +0x1b7
  go.etcd.io/etcd/mvcc.(*treeIndex).Revisions()
      /go/src/go.etcd.io/etcd/mvcc/index.go:115 +0x20d
  go.etcd.io/etcd/mvcc.(*storeTxnRead).rangeKeys()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:128 +0x37a
  go.etcd.io/etcd/mvcc.(*storeTxnRead).Range()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:50 +0xe4
  go.etcd.io/etcd/mvcc.(*txnReadWrite).Range()
      <autogenerated>:1 +0xf4
  go.etcd.io/etcd/mvcc.(*metricsTxnWrite).Range()
      /go/src/go.etcd.io/etcd/mvcc/metrics_txn.go:36 +0x12a
  go.etcd.io/etcd/etcdserver.(*applierV3backend).Range()
      /go/src/go.etcd.io/etcd/etcdserver/apply.go:275 +0x2fc
  go.etcd.io/etcd/etcdserver.(*EtcdServer).Range.func3()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:104 +0x9d
  go.etcd.io/etcd/etcdserver.(*EtcdServer).doSerialize()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:554 +0xad
  go.etcd.io/etcd/etcdserver.(*EtcdServer).Range()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:105 +0x225
  go.etcd.io/etcd/etcdserver/api/v3rpc.(*kvServer).Range()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/key.go:52 +0xb0
  go.etcd.io/etcd/etcdserver/api/v3rpc.(*quotaKVServer).Range()
      <autogenerated>:1 +0x87
  go.etcd.io/etcd/etcdserver/etcdserverpb._KV_Range_Handler.func1()
      /go/src/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3473 +0xa1
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:31 +0x2d3
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).UnaryServerInterceptor.func1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server_metrics.go:107 +0xff
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34 +0x1d9
  go.etcd.io/etcd/etcdserver/api/v3rpc.newUnaryInterceptor.func1()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/interceptor.go:60 +0xf0
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34 +0x1d9
  go.etcd.io/etcd/etcdserver/api/v3rpc.newLogUnaryInterceptor.func1()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/interceptor.go:67 +0xda
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:39 +0x33d
  go.etcd.io/etcd/etcdserver/etcdserverpb._KV_Range_Handler()
      /go/src/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3475 +0x1d8
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).processUnaryRPC()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:1026 +0xa8b
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleStream()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:1252 +0x138d
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:699 +0xac
Goroutine 1012 (running) created at:
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:697 +0xb8
  go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport/http2_server.go:432 +0x14cb
  go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport/http2_server.go:472 +0x37a
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:695 +0x170
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleRawConn.func2()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:649 +0x53
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleRawConn.func3()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:657 +0x34
Goroutine 843 (finished) created at:
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:697 +0xb8
  go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport/http2_server.go:432 +0x14cb
  go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport/http2_server.go:472 +0x37a
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:695 +0x170
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleRawConn.func2()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:649 +0x53
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleRawConn.func3()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:657 +0x34
==================
--- FAIL: TestBarrierSingleNode (0.25s)
    testing.go:771: race detected during execution of test
=== RUN   TestBarrierMultiNode
==================
WARNING: DATA RACE
Write at 0x00c00042d4b8 by goroutine 689:
  go.etcd.io/etcd/mvcc.(*keyIndex).put()
      /go/src/go.etcd.io/etcd/mvcc/key_index.go:101 +0x206
  go.etcd.io/etcd/mvcc.(*keyIndex).tombstone()
      /go/src/go.etcd.io/etcd/mvcc/key_index.go:141 +0x2ea
  go.etcd.io/etcd/mvcc.(*treeIndex).Tombstone()
      /go/src/go.etcd.io/etcd/mvcc/index.go:151 +0x19c
  go.etcd.io/etcd/mvcc.(*storeTxnWrite).delete()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:283 +0x375
  go.etcd.io/etcd/mvcc.(*storeTxnWrite).deleteRange()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:251 +0x1b9
  go.etcd.io/etcd/mvcc.(*storeTxnWrite).DeleteRange()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:90 +0x81
  go.etcd.io/etcd/mvcc.(*metricsTxnWrite).DeleteRange()
      /go/src/go.etcd.io/etcd/mvcc/metrics_txn.go:41 +0xe1
  go.etcd.io/etcd/mvcc.(*watchableStoreTxnWrite).DeleteRange()
      <autogenerated>:1 +0xb1
  go.etcd.io/etcd/etcdserver.(*applierV3backend).DeleteRange()
      /go/src/go.etcd.io/etcd/etcdserver/apply.go:244 +0x241
  go.etcd.io/etcd/etcdserver.(*quotaApplierV3).DeleteRange()
      <autogenerated>:1 +0x87
  go.etcd.io/etcd/etcdserver.(*authApplierV3).DeleteRange()
      /go/src/go.etcd.io/etcd/etcdserver/apply_auth.go:104 +0x1ce
  go.etcd.io/etcd/etcdserver.(*applierV3backend).Apply()
      /go/src/go.etcd.io/etcd/etcdserver/apply.go:126 +0x4dd
  go.etcd.io/etcd/etcdserver.(*quotaApplierV3).Apply()
      <autogenerated>:1 +0x68
  go.etcd.io/etcd/etcdserver.(*authApplierV3).Apply()
      /go/src/go.etcd.io/etcd/etcdserver/apply_auth.go:58 +0xf3
  go.etcd.io/etcd/etcdserver.(*EtcdServer).applyEntryNormal()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:2011 +0x3c8
  go.etcd.io/etcd/etcdserver.(*EtcdServer).apply()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:1925 +0x77e
  go.etcd.io/etcd/etcdserver.(*EtcdServer).applyEntries()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:1349 +0x175
  go.etcd.io/etcd/etcdserver.(*EtcdServer).applyAll()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:1075 +0x98
  go.etcd.io/etcd/etcdserver.(*EtcdServer).run.func8()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:1020 +0x53
  go.etcd.io/etcd/pkg/schedule.(*fifo).run()
      /go/src/go.etcd.io/etcd/pkg/schedule/schedule.go:157 +0x105
Previous read at 0x00c00042d4b8 by goroutine 271:
  go.etcd.io/etcd/mvcc.(*keyIndex).findGeneration()
      /go/src/go.etcd.io/etcd/mvcc/key_index.go:313 +0x150
  go.etcd.io/etcd/mvcc.(*keyIndex).get()
      /go/src/go.etcd.io/etcd/mvcc/key_index.go:160 +0x255
  go.etcd.io/etcd/mvcc.(*treeIndex).Revisions.func1()
      /go/src/go.etcd.io/etcd/mvcc/index.go:116 +0xab
  go.etcd.io/etcd/mvcc.(*treeIndex).visit.func1()
      /go/src/go.etcd.io/etcd/mvcc/index.go:102 +0x85
  go.etcd.io/etcd/vendor/github.com/google/btree.(*node).iterate()
      /go/src/go.etcd.io/etcd/vendor/github.com/google/btree/btree.go:523 +0x17f
  go.etcd.io/etcd/vendor/github.com/google/btree.(*BTree).AscendGreaterOrEqual()
      /go/src/go.etcd.io/etcd/vendor/github.com/google/btree/btree.go:759 +0xa9
  go.etcd.io/etcd/mvcc.(*treeIndex).visit()
      /go/src/go.etcd.io/etcd/mvcc/index.go:98 +0x3d9
  go.etcd.io/etcd/mvcc.(*treeIndex).Revisions()
      /go/src/go.etcd.io/etcd/mvcc/index.go:115 +0x20d
  go.etcd.io/etcd/mvcc.(*storeTxnRead).rangeKeys()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:128 +0x37a
  go.etcd.io/etcd/mvcc.(*storeTxnRead).Range()
      /go/src/go.etcd.io/etcd/mvcc/kvstore_txn.go:50 +0xe4
  go.etcd.io/etcd/mvcc.(*txnReadWrite).Range()
      <autogenerated>:1 +0xf4
  go.etcd.io/etcd/mvcc.(*metricsTxnWrite).Range()
      /go/src/go.etcd.io/etcd/mvcc/metrics_txn.go:36 +0x12a
  go.etcd.io/etcd/etcdserver.(*applierV3backend).Range()
      /go/src/go.etcd.io/etcd/etcdserver/apply.go:275 +0x2fc
  go.etcd.io/etcd/etcdserver.(*EtcdServer).Range.func3()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:104 +0x9d
  go.etcd.io/etcd/etcdserver.(*EtcdServer).doSerialize()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:554 +0xad
  go.etcd.io/etcd/etcdserver.(*EtcdServer).Range()
      /go/src/go.etcd.io/etcd/etcdserver/v3_server.go:105 +0x225
  go.etcd.io/etcd/etcdserver/api/v3rpc.(*kvServer).Range()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/key.go:52 +0xb0
  go.etcd.io/etcd/etcdserver/api/v3rpc.(*quotaKVServer).Range()
      <autogenerated>:1 +0x87
  go.etcd.io/etcd/etcdserver/etcdserverpb._KV_Range_Handler.func1()
      /go/src/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3473 +0xa1
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:31 +0x2d3
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).UnaryServerInterceptor.func1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server_metrics.go:107 +0xff
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34 +0x1d9
  go.etcd.io/etcd/etcdserver/api/v3rpc.newUnaryInterceptor.func1()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/interceptor.go:60 +0xf0
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34 +0x1d9
  go.etcd.io/etcd/etcdserver/api/v3rpc.newLogUnaryInterceptor.func1()
      /go/src/go.etcd.io/etcd/etcdserver/api/v3rpc/interceptor.go:67 +0xda
  go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1()
      /go/src/go.etcd.io/etcd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:39 +0x33d
  go.etcd.io/etcd/etcdserver/etcdserverpb._KV_Range_Handler()
      /go/src/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3475 +0x1d8
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).processUnaryRPC()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:1026 +0xa8b
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleStream()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:1252 +0x138d
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:699 +0xac
Goroutine 689 (running) created at:
  go.etcd.io/etcd/pkg/schedule.NewFIFOScheduler()
      /go/src/go.etcd.io/etcd/pkg/schedule/schedule.go:70 +0x2b1
  go.etcd.io/etcd/etcdserver.(*EtcdServer).run()
      /go/src/go.etcd.io/etcd/etcdserver/server.go:905 +0x33f
Goroutine 271 (finished) created at:
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:697 +0xb8
  go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport/http2_server.go:432 +0x14cb
  go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/internal/transport/http2_server.go:472 +0x37a
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).serveStreams()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:695 +0x170
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleRawConn.func2()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:649 +0x53
  go.etcd.io/etcd/vendor/google.golang.org/grpc.(*Server).handleRawConn.func3()
      /go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/server.go:657 +0x34
==================
--- FAIL: TestBarrierMultiNode (0.66s)
    testing.go:771: race detected during execution of test

@xiang90
Copy link
Contributor Author

xiang90 commented May 14, 2019 

--- FAIL: proxy-clientv3/integration
--- FAIL: tests/e2e-proxy
make: *** [docker-test-coverage] Error 1
travis_time:end:2b992fcc:start=1555527843145227846,finish=1555530481575826563,duration=2638430598717
�[0K�[31;1mThe command "case "${TARGET}" in
  linux-amd64-fmt)
    docker run --rm \
      --volume=`pwd`:/go/src/go.etcd.io/etcd gcr.io/etcd-development/etcd-test:go${TRAVIS_GO_VERSION} \
      /bin/bash -c "GOARCH=amd64 PASSES='fmt bom dep' ./test"
    ;;
  linux-amd64-integration-1-cpu)
    docker run --rm \
      --volume=`pwd`:/go/src/go.etcd.io/etcd gcr.io/etcd-development/etcd-test:go${TRAVIS_GO_VERSION} \
      /bin/bash -c "GOARCH=amd64 CPU=1 PASSES='integration' ./test"
    ;;
  linux-amd64-integration-2-cpu)
    docker run --rm \
      --volume=`pwd`:/go/src/go.etcd.io/etcd gcr.io/etcd-development/etcd-test:go${TRAVIS_GO_VERSION} \
      /bin/bash -c "GOARCH=amd64 CPU=2 PASSES='integration' ./test"
    ;;
  linux-amd64-integration-4-cpu)
    docker run --rm \
      --volume=`pwd`:/go/src/go.etcd.io/etcd gcr.io/etcd-development/etcd-test:go${TRAVIS_GO_VERSION} \
      /bin/bash -c "GOARCH=amd64 CPU=4 PASSES='integration' ./test"
    ;;
  linux-amd64-functional)
    docker run --rm \
      --volume=`pwd`:/go/src/go.etcd.io/etcd gcr.io/etcd-development/etcd-test:go${TRAVIS_GO_VERSION} \
      /bin/bash -c "./build && GOARCH=amd64 PASSES='functional' ./test"
    ;;
  linux-amd64-unit)
    docker run --rm \
      --volume=`pwd`:/go/src/go.etcd.io/etcd gcr.io/etcd-development/etcd-test:go${TRAVIS_GO_VERSION} \
      /bin/bash -c "GOARCH=amd64 PASSES='unit' ./test"
    ;;
  all-build)
    docker run --rm \
      --volume=`pwd`:/go/src/go.etcd.io/etcd gcr.io/etcd-development/etcd-test:go${TRAVIS_GO_VERSION} \
      /bin/bash -c "GOARCH=amd64 PASSES='build' ./test \
        && GOARCH=386 PASSES='build' ./test \
        && GO_BUILD_FLAGS='-v' GOOS=darwin GOARCH=amd64 ./build \
        && GO_BUILD_FLAGS='-v' GOOS=windows GOARCH=amd64 ./build \
        && GO_BUILD_FLAGS='-v' GOARCH=arm ./build \
        && GO_BUILD_FLAGS='-v' GOARCH=arm64 ./build \
        && GO_BUILD_FLAGS='-v' GOARCH=ppc64le ./build"
    ;;
  linux-amd64-grpcproxy)
    sudo HOST_TMP_DIR=/tmp TEST_OPTS="PASSES='build grpcproxy'" make docker-test
    ;;
  linux-amd64-coverage)
    sudo HOST_TMP_DIR=/tmp make docker-test-coverage
    ;;
  linux-amd64-fmt-unit-go-tip)
    GOARCH=amd64 PASSES='fmt unit' ./test
    ;;
  linux-386-unit)
    docker run --rm \
      --volume=`pwd`:/go/src/go.etcd.io/etcd gcr.io/etcd-development/etcd-test:go${TRAVIS_GO_VERSION} \
      /bin/bash -c "GOARCH=386 PASSES='unit' ./test"
    ;;
esac
" exited with 2.�[0m

@xiang90
Copy link
Contributor Author

xiang90 commented May 15, 2019 

=== RUN   TestLeasingTxnOwnerIf
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x60 pc=0xaf8a8d]
goroutine 20358 [running]:
google.golang.org/grpc.(*csAttempt).sendMsg(0xc0017386e0, 0x12b7d20, 0xc0011ba620, 0xc00615e92b, 0x5, 0x5, 0xc00615e920, 0xb, 0xb, 0xc00615e920, ...)
	/go/pkg/mod/google.golang.org/grpc@v1.14.0/stream.go:713 +0x14d
google.golang.org/grpc.(*clientStream).SendMsg.func2(0xc0017386e0, 0x48, 0x48)
	/go/pkg/mod/google.golang.org/grpc@v1.14.0/stream.go:637 +0x186
google.golang.org/grpc.(*clientStream).withRetry(0xc0000bed80, 0xc001beaeb0, 0xc001781aa0, 0xc00615e92b, 0x5)
	/go/pkg/mod/google.golang.org/grpc@v1.14.0/stream.go:530 +0x431
google.golang.org/grpc.(*clientStream).SendMsg(0xc0000bed80, 0x12b7d20, 0xc0011ba620, 0x0, 0x0)
	/go/pkg/mod/google.golang.org/grpc@v1.14.0/stream.go:643 +0x80c
go.etcd.io/etcd/v3/etcdserver/etcdserverpb.(*watchWatchClient).Send(0xc0011ba5f0, 0xc0011ba620, 0x0, 0x1cdfb18)
	/go/src/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3689 +0x6b
go.etcd.io/etcd/v3/clientv3.(*watchGrpcStream).run(0xc001757450)
	/go/src/go.etcd.io/etcd/clientv3/watch.go:546 +0xd8d
created by go.etcd.io/etcd/v3/clientv3.(*watcher).newWatcherGrpcStream
	/go/src/go.etcd.io/etcd/clientv3/watch.go:277 +0x58c

@xiang90
Copy link
Contributor Author

xiang90 commented May 15, 2019 

--- FAIL: TestBalancerUnderNetworkPartitionWatchLeader (3.43s)
    network_partition_test.go:266: took too long to detect leader lost

@xiang90
Copy link
Contributor Author

xiang90 commented May 15, 2019 

--- FAIL: TestLeasingDeleteRangeContendTxn (0.73s)
    testing.go:771: race detected during execution of test

@jingyih jingyih mentioned this issue May 29, 2019
Merged
@xiang90
Copy link
Contributor Author

xiang90 commented May 31, 2019

--- FAIL: TestBalancerUnderNetworkPartitionWatchLeader (3.29s)
network_partition_test.go:266: took too long to detect leader lost

https://travis-ci.com/etcd-io/etcd/jobs/204590472

@spzala
Copy link
Member

spzala commented Jun 4, 2019 •
edited
Loading

Noticed these failures in two different PRs today (basically same failure point for both tests):

--- FAIL: TestV3CurlAuth (1.00s)
    v3_curl_test.go:203: failed testV3CurlAuth add user with curl (read /dev/ptmx: input/output error (expected "revision", got ["curl: (52) Empty reply from server\r\n"]))
    ctl_v3_test.go:168: error closing etcd processes (exit status 2)

--- FAIL: TestV3CurlAuthClientTLSCertAuth (0.26s)
    v3_curl_test.go:203: failed testV3CurlAuth add user with curl (read /dev/ptmx: input/output error (expected "revision", got ["curl: (56) Unexpected EOF\r\n"]))
    ctl_v3_test.go:168: error closing etcd processes (exit status 2)

https://semaphoreci.com/etcd-io/etcd/branches/pull-request-10631/builds/14 for 386-e2e

@spzala spzala mentioned this issue Jun 4, 2019
Merged
@spzala
Copy link
Member

spzala commented Jun 5, 2019 

=== RUN   TestLeasingReconnectTxn
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x60 pc=0xb5c7ca]
goroutine 30639 [running]:
go.etcd.io/etcd/vendor/google.golang.org/grpc.(*csAttempt).sendMsg(0xc000150210, 0x13438c0, 0xc0011b7760, 0xc0016dc2f8, 0x5, 0x5, 0xc0016dc300, 0xb, 0xb, 0xc0016dc300, ...)
	/go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/stream.go:713 +0x14a
go.etcd.io/etcd/vendor/google.golang.org/grpc.(*clientStream).SendMsg.func2(0xc000150210, 0xb5b88f, 0xc00283ccd0)
	/go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/stream.go:637 +0x186
go.etcd.io/etcd/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc00017a5a0, 0xc00283ccd0, 0xc0016abaa8, 0xc0016dc2f8, 0x0)
	/go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/stream.go:530 +0x43d
go.etcd.io/etcd/vendor/google.golang.org/grpc.(*clientStream).SendMsg(0xc00017a5a0, 0x13438c0, 0xc0011b7760, 0x0, 0x0)
	/go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/stream.go:643 +0x777
go.etcd.io/etcd/etcdserver/etcdserverpb.(*watchWatchClient).Send(0xc0011b7730, 0xc0011b7760, 0x0, 0x1e4cf98)
	/go/src/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3697 +0x6b
go.etcd.io/etcd/clientv3.(*watchGrpcStream).run(0xc000360410)
	/go/src/go.etcd.io/etcd/clientv3/watch.go:546 +0xd9b
created by go.etcd.io/etcd/clientv3.(*watcher).newWatcherGrpcStream
	/go/src/go.etcd.io/etcd/clientv3/watch.go:277 +0x58c

https://travis-ci.com/etcd-io/etcd/jobs/205678389#L2183

@jingyih
Copy link
Contributor

jingyih commented Jun 7, 2019

Noticed these failures in two different PRs today (basically same failure point for both tests):

--- FAIL: TestV3CurlAuth (1.00s)
    v3_curl_test.go:203: failed testV3CurlAuth add user with curl (read /dev/ptmx: input/output error (expected "revision", got ["curl: (52) Empty reply from server\r\n"]))
    ctl_v3_test.go:168: error closing etcd processes (exit status 2)

--- FAIL: TestV3CurlAuthClientTLSCertAuth (0.26s)
    v3_curl_test.go:203: failed testV3CurlAuth add user with curl (read /dev/ptmx: input/output error (expected "revision", got ["curl: (56) Unexpected EOF\r\n"]))
    ctl_v3_test.go:168: error closing etcd processes (exit status 2)

https://semaphoreci.com/etcd-io/etcd/branches/pull-request-10631/builds/14 for 386-e2e

Fixed by #10800.

@jingyih jingyih mentioned this issue Jun 14, 2019
Merged
@jingyih
Copy link
Contributor

jingyih commented Jun 14, 2019

=== RUN TestLeasingGetWithOpts
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x60 pc=0xb5c7ca]
goroutine 24216 [running]:
go.etcd.io/etcd/vendor/google.golang.org/grpc.(*csAttempt).sendMsg(0xc00065c160, 0x1346c20, 0xc0003b0ec0, 0xc002edc378, 0x5, 0x5, 0xc002edc380, 0xb, 0xb, 0xc002edc380, ...)
/go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/stream.go:713 +0x14a
go.etcd.io/etcd/vendor/google.golang.org/grpc.(*clientStream).SendMsg.func2(0xc00065c160, 0xb5b88f, 0xc00016f130)
/go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/stream.go:637 +0x186
go.etcd.io/etcd/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc00183a240, 0xc00016f130, 0xc0003edaa8, 0xc002edc378, 0x0)
/go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/stream.go:530 +0x43d
go.etcd.io/etcd/vendor/google.golang.org/grpc.(*clientStream).SendMsg(0xc00183a240, 0x1346c20, 0xc0003b0ec0, 0x0, 0x0)
/go/src/go.etcd.io/etcd/vendor/google.golang.org/grpc/stream.go:643 +0x777
go.etcd.io/etcd/etcdserver/etcdserverpb.(*watchWatchClient).Send(0xc00088cbb0, 0xc0003b0ec0, 0x0, 0x1e53018)
/go/src/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3697 +0x6b
go.etcd.io/etcd/clientv3.(*watchGrpcStream).run(0xc0001275f0)
/go/src/go.etcd.io/etcd/clientv3/watch.go:546 +0xd9b
created by go.etcd.io/etcd/clientv3.(*watcher).newWatcherGrpcStream
/go/src/go.etcd.io/etcd/clientv3/watch.go:277 +0x58c

@jingyih
Copy link
Contributor

jingyih commented Jun 18, 2019

This one is pretty new, only observed it in two very recent tests (might be related to recently merged PRs?)
https://travis-ci.com/etcd-io/etcd/jobs/208110433#L1176
https://travis-ci.com/etcd-io/etcd/jobs/208860886#L1256

--- FAIL: TestKVPutError (0.18s)
kv_test.go:56: expected etcdserver: request is too large, got etcdserver: mvcc: database space exceeded

@spzala
Copy link
Member

spzala commented Jun 30, 2019

As reported under #10787 (I believe we should close it and track the test failure here)

=== RUN   TestCtlV3AuthEndpointHealth
panic: test timed out after 20m0s

goroutine 6839 [running]:
testing.(*M).startAlarm.func1()
	/usr/local/go/src/testing/testing.go:1334 +0xdf
created by time.goFunc
	/usr/local/go/src/time/sleep.go:169 +0x44

goroutine 1 [chan receive]:
testing.(*T).Run(0xc0001ca100, 0xcba649, 0x1b, 0xce1508, 0x4b6c01)
	/usr/local/go/src/testing/testing.go:917 +0x381
make: *** [docker-test] Error 1

@spzala spzala mentioned this issue Jun 30, 2019
Closed
@tbg tbg mentioned this issue Jul 9, 2019
Closed
tbg added a commit to tbg/etcd that referenced this issue Jul 11, 2019
@tbg
test: allow failures in linux-amd64-integration-4-cpu
b7327b1 
This run *should* certainly pass, but it's consistently the one that
fails with a regularity that essentially blocks the CI pipeline.

Someone needs to take a look at etcd-io#10700, but in the meantime, the show
must go on.
@tbg tbg mentioned this issue Jul 11, 2019
Merged
@tbg
Copy link
Contributor

tbg commented Jul 11, 2019

Just as a heads up, the linux-amd64-integration-4-cpu is now skipped as a temporary measure. From what I saw, it is usually the target that incurred the failures.

In my book, this also increases the priority of resolving these flakes.

@tbg
Copy link
Contributor

tbg commented Jul 17, 2019 •
edited by jingyih
Loading

@jingyih in response to #10700 (comment) these errors still appear today: https://semaphoreci.com/etcd-io/etcd/branches/pull-request-10903/builds/2

--- FAIL: TestCtlV2BackupSnapshot (0.46s)
    ctl_v2_test.go:536: could not start etcd process cluster (read /dev/ptmx: input/output error)
=== RUN   TestCtlV2BackupV3
--- PASS: TestCtlV2BackupV3 (2.34s)
=== RUN   TestCtlV2BackupV3Snapshot
--- FAIL: TestCtlV2BackupV3Snapshot (0.31s)
    ctl_v2_test.go:536: could not start etcd process cluster (read /dev/ptmx: input/output error)

I found this issue which is perhaps related; creack/pty#21

Edit (jingyih@, Fri 19 Jul 2019 05:47:20 PM PDT): fixed by #10908.

@jingyih
Copy link
Contributor

jingyih commented Jul 17, 2019

@tbg This is not the same as those previously fixed by #10800. They may be related and we should investigate. Thanks for reporting!

@tbg
Copy link
Contributor

tbg commented Jul 19, 2019

^- I pulled on this thread because these tests seem to somehow be related to version compatibility, and found that I broke this #10889. I probably didn't notice because the semaphore tests were failing for unrelated reasons.

Will investigate more. I assume there's a better error message that just gets lost somehow.

PS I am using HOST_TMP_DIR=/tmp TEST_OPTS="TESTCASE='TestCtlV2Backup.*Snap' GOARCH=386 PASSES='build e2e'" make docker-test on my linux machine to verify.

@tbg
Copy link
Contributor

tbg commented Jul 19, 2019

Ha, took me 45 minutes to figure out how to get the output:

HOST_TMP_DIR=/tmp TEST_OPTS="EXPECT_DEBUG=true CPU=4 TESTCASE='TestCtlV2BackupV3' GOARCH=386 PASSES='e2e'" make docker-test

Now that I have it, here we go:

../../bin/etcd-12937: 2019-07-19 12:23:08.623487 I | etcdserver: 6c0a2f9e2301 as single-node; fast-forwarding 9 ticks (election ticks 10)
../../bin/etcd-12937: panic: removed all voters
../../bin/etcd-12937:
../../bin/etcd-12937: goroutine 141 [running]:
../../bin/etcd-12937: go.etcd.io/etcd/raft.(*raft).applyConfChange(0xb4ee000, 0x0, 0x0, 0x1, 0x7181d758, 0xca50e935, 0x0, 0x0, 0x0, 0x0, ...)
../../bin/etcd-12937: 	/go/src/go.etcd.io/etcd/raft/raft.go:1406 +0x43f
../../bin/etcd-12937: go.etcd.io/etcd/raft.(*node).run(0xb49c180, 0xb4ee000)
../../bin/etcd-12937: 	/go/src/go.etcd.io/etcd/raft/node.go:372 +0xcd6
../../bin/etcd-12937: created by go.etcd.io/etcd/raft.RestartNode
../../bin/etcd-12937: 	/go/src/go.etcd.io/etcd/raft/node.go:258 +0x252

This indeed introduced in #10889. Now to find out whether the test is doing something silly.

@tbg
Copy link
Contributor

tbg commented Jul 19, 2019

Still poking, but confirming that at least from the vantage point of raft there's a voter being removed:

../../bin/etcd-15895: {"level":"info","ts":"2019-07-19T13:02:09.719Z","caller":"raft/node.go:344","msg":"applying conf change {ID:0 Type:ConfChangeRemoveNode NodeID:14578408409545168728 Context:[] XXX_unrecognized:[]}, current config is voters=(14578408409545168728)"}

@tbg
Copy link
Contributor

tbg commented Jul 19, 2019

Ok I got it, this shady behavior on the part of etcd. There's this code in restartAsStandaloneNode, a method, which if the name suggests correctly, basically wants to rewrite the config of a node and make it the only voter in it.

etcd/etcdserver/raft.go

Lines 573 to 580 in 905aefc

// force append the configuration change entries
toAppEnts := createConfigChangeEnts(
cfg.Logger,
getIDs(cfg.Logger, snapshot, ents),
uint64(id),
st.Term,
st.Commit,
)

Unfortunately that code removes before it adds, so if our node is B and the config is A, then it will first remove A and then add B, ending up with something empty in the middle. It needs to add everyone first, then remove. I'll send a patch.

tbg added a commit to tbg/etcd that referenced this issue Jul 19, 2019
@tbg
etcdserver: fix createConfChangeEnts
ed8b070 
It created a sequence of conf changes that could intermittently cause an
empty set of voters, which Raft asserts against as of etcd-io#10889.

This fixes TestCtlV2BackupSnapshot and TestCtlV2BackupV3Snapshot, see:
etcd-io#10700 (comment)
tbg added a commit to tbg/etcd that referenced this issue Jul 19, 2019
@tbg
etcdserver: fix createConfChangeEnts
a30bc1e 
It created a sequence of conf changes that could intermittently cause an
empty set of voters, which Raft asserts against as of etcd-io#10889.

This fixes TestCtlV2BackupSnapshot and TestCtlV2BackupV3Snapshot, see:
etcd-io#10700 (comment)
@tbg tbg mentioned this issue Jul 19, 2019
Merged
@tbg
Copy link
Contributor

tbg commented Jul 19, 2019

#10908

tbg added a commit to tbg/etcd that referenced this issue Jul 19, 2019
@tbg
etcdserver: fix createConfChangeEnts
94d61ef 
It created a sequence of conf changes that could intermittently cause an
empty set of voters, which Raft asserts against as of etcd-io#10889.

This fixes TestCtlV2BackupSnapshot and TestCtlV2BackupV3Snapshot, see:
etcd-io#10700 (comment)
tbg added a commit to tbg/etcd that referenced this issue Jul 19, 2019
@tbg
etcdserver: fix createConfChangeEnts
eb4d9b6 
It created a sequence of conf changes that could intermittently cause an
empty set of voters, which Raft asserts against as of etcd-io#10889.

This fixes TestCtlV2BackupSnapshot and TestCtlV2BackupV3Snapshot, see:
etcd-io#10700 (comment)
@gyuho gyuho mentioned this issue Jul 26, 2019
Closed
19 tasks
@gyuho gyuho mentioned this issue Aug 5, 2019
Closed
@gyuho
Copy link
Contributor

gyuho commented Aug 5, 2019

Let's continue this on #10979

@gyuho gyuho closed this as completed Aug 5, 2019
@kennytm kennytm mentioned this issue Jan 9, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
good first issue help wanted
Milestone
No milestone
Development

No branches or pull requests
6 participants
@xiang90 @tbg @spzala @gyuho @jingyih @wenjiaswe