Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

etcd perforamance limit #9112

Closed
WanLinghao opened this issue Jan 8, 2018 · 12 comments
Closed

etcd perforamance limit #9112

WanLinghao opened this issue Jan 8, 2018 · 12 comments

Comments

@WanLinghao
Copy link

WanLinghao commented Jan 8, 2018

We are doing a etcd performance limit test recently.The test goes like this:
Insert some data into an etcd cluster, then test its performance, such as read time consume.
At first, we want to insert 500,0000 data into the cluster.But the cluster collapsed at 50,0000 data during test process.
The test data is all about kubernetes info, all data follows the pattern below

key: /registry/pod/KKR5IS/Dihxg1mav 
value:
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-01-02T02:46:14Z"},"level":"Metadata","t    imestamp":"2018-01-02T02:46:14Z","auditID":"'23c10956-3479-41af-af1f-958f4081e351","stage":"RequestReceived","requestURI":"/apis/rbac.authorization.k8s.io/v1/rolebindings?limit=500\u0026resourceVersion=0","verb    ":"list","user":{"username":"system:apiserver","uid":"23c10956-3479-41af-af1f-958f4081e351","groups":["system:masters"]},"sourceIPs":["127.0.0.1"],    "objectRef":{"resource":"rolebindings","apiGroup":"rbac.authorization.k8s.io","apiVersion":"v1"},"requestReceivedTimestamp":"2018-01-02T02:46:14.43    9794Z","stageTimestamp":"2018-01-02T02:46:14.439794Z"}

We want to know where's the limit of etcd cluster? Did we hit the limit of etcd cluster?
If not, anything wrong we did?

Here's some information maybe useful.
etcd Version: 3.1.3
3-node cluster, each node has 4GB memory.

@xiang90
Copy link
Contributor

xiang90 commented Jan 8, 2018

We successfully simulated 5Mi pods case. It seems to me that you did something wrong.

Please share the scripts(etcd setup, put test) to reproduce your test, and please share the stack trace for the crash.

@WanLinghao
Copy link
Author

@xiang90 Thanks for your reply
1.What do you mean about 5Mi pods case? five million pods?

2.Here's some info maybe useful
stack:

2018-01-08 19:42:49.076911 W | etcdserver: apply entries took too long [567.044466ms for 1 entries]
2018-01-08 19:42:49.076942 W | etcdserver: avoid queries with large range/delete range!
2018-01-08 19:42:58.261235 W | rafthttp: the clock difference against peer eab186b4ce5cf0eb is too high [5h1m15.779133315s > 1s]
2018-01-08 19:43:03.298594 W | etcdserver: apply entries took too long [336.357427ms for 1 entries]
2018-01-08 19:43:03.298612 W | etcdserver: avoid queries with large range/delete range!
2018-01-08 19:43:13.591300 W | rafthttp: the clock difference against peer 934a1c4495cff6bc is too high [1m14.936892866s > 1s]
2018-01-08 19:43:28.261337 W | rafthttp: the clock difference against peer eab186b4ce5cf0eb is too high [5h1m15.801239336s > 1s]
2018-01-08 19:43:43.591411 W | rafthttp: the clock difference against peer 934a1c4495cff6bc is too high [1m14.838990689s > 1s]
2018-01-08 19:43:53.448065 W | etcdserver: apply entries took too long [271.009491ms for 1 entries]
2018-01-08 19:43:53.448083 W | etcdserver: avoid queries with large range/delete range!
2018-01-08 19:43:58.268147 W | rafthttp: the clock difference against peer eab186b4ce5cf0eb is too high [5h1m15.809444743s > 1s]
2018-01-08 19:43:58.428966 W | etcdserver: apply entries took too long [133.439744ms for 1 entries]
2018-01-08 19:43:58.428982 W | etcdserver: avoid queries with large range/delete range!
2018-01-08 19:44:13.626595 W | rafthttp: the clock difference against peer 934a1c4495cff6bc is too high [1m14.845869235s > 1s]
2018-01-08 19:44:28.268269 W | rafthttp: the clock difference against peer eab186b4ce5cf0eb is too high [5h1m15.808967744s > 1s]
2018-01-08 19:44:43.630542 W | rafthttp: the clock difference against peer 934a1c4495cff6bc is too high [1m14.872461428s > 1s]
2018-01-08 19:44:58.268367 W | rafthttp: the clock difference against peer eab186b4ce5cf0eb is too high [5h1m15.808151555s > 1s]
2018-01-08 19:45:13.630816 W | rafthttp: the clock difference against peer 934a1c4495cff6bc is too high [1m14.836763166s > 1s]
2018-01-08 19:45:28.268560 W | rafthttp: the clock difference against peer eab186b4ce5cf0eb is too high [5h1m15.812235493s > 1s]
2018-01-08 19:45:32.823395 W | etcdserver: apply entries took too long [408.698005ms for 1 entries]
2018-01-08 19:45:32.823426 W | etcdserver: avoid queries with large range/delete range!
2018-01-08 19:45:42.784484 W | etcdserver: apply entries took too long [105.287791ms for 1 entries]
2018-01-08 19:45:42.784515 W | etcdserver: avoid queries with large range/delete range!
2018-01-08 19:45:43.651048 W | rafthttp: the clock difference against peer 934a1c4495cff6bc is too high [1m14.839488164s > 1s]
2018-01-08 19:45:49.767035 W | wal: sync duration of 1.797296637s, expected less than 1s
2018-01-08 19:45:55.806348 W | etcdserver: apply entries took too long [139.797957ms for 1 entries]
2018-01-08 19:45:55.806370 W | etcdserver: avoid queries with large range/delete range!
2018-01-08 19:45:58.268690 W | rafthttp: the clock difference against peer eab186b4ce5cf0eb is too high [5h1m15.812586645s > 1s]
2018-01-08 19:45:58.857503 W | etcdserver: apply entries took too long [453.311067ms for 1 entries]
2018-01-08 19:45:58.857526 W | etcdserver: avoid queries with large range/delete range!
2018-01-08 19:46:03.710107 W | etcdserver: apply entries took too long [280.215967ms for 1 entries]
2018-01-08 19:46:03.710130 W | etcdserver: avoid queries with large range/delete range!
2018-01-08 19:46:13.651172 W | rafthttp: the clock difference against peer 934a1c4495cff6bc is too high [1m14.840167418s > 1s]
fatal error: runtime: out of memory

runtime stack:
runtime.throw(0x56208e31fc1f, 0x16)
	/usr/lib/golang/src/runtime/panic.go:596 +0x97 fp=0x7fdc669adc18 sp=0x7fdc669adbf8
runtime.sysMap(0xc5635c0000, 0x19b00000, 0x56208db07b00, 0x56208ec8d758)
	/usr/lib/golang/src/runtime/mem_linux.go:216 +0x1d2 fp=0x7fdc669adc58 sp=0x7fdc669adc18
runtime.(*mheap).sysAlloc(0x56208ec74860, 0x19b00000, 0x7fdc669add10)
	/usr/lib/golang/src/runtime/malloc.go:428 +0x376 fp=0x7fdc669adce0 sp=0x7fdc669adc58
runtime.(*mheap).grow(0x56208ec74860, 0xcd7f, 0x0)
	/usr/lib/golang/src/runtime/mheap.go:774 +0x64 fp=0x7fdc669add38 sp=0x7fdc669adce0
runtime.(*mheap).allocSpanLocked(0x56208ec74860, 0xcd7f, 0x7fd9de9a48d8)
	/usr/lib/golang/src/runtime/mheap.go:678 +0x451 fp=0x7fdc669add78 sp=0x7fdc669add38
runtime.(*mheap).alloc_m(0x56208ec74860, 0xcd7f, 0x100000000, 0x0)
	/usr/lib/golang/src/runtime/mheap.go:562 +0xe6 fp=0x7fdc669addb0 sp=0x7fdc669add78
runtime.(*mheap).alloc.func1()
	/usr/lib/golang/src/runtime/mheap.go:627 +0x4d fp=0x7fdc669adde8 sp=0x7fdc669addb0
runtime.systemstack(0x7fdc669ade08)
	/usr/lib/golang/src/runtime/asm_amd64.s:343 +0xa1 fp=0x7fdc669addf0 sp=0x7fdc669adde8
runtime.(*mheap).alloc(0x56208ec74860, 0xcd7f, 0x10100000000, 0x56208dacfa74)
	/usr/lib/golang/src/runtime/mheap.go:628 +0xa2 fp=0x7fdc669ade40 sp=0x7fdc669addf0
runtime.largeAlloc(0x19afdfe3, 0x56208db01c01, 0xc530e7e000)
	/usr/lib/golang/src/runtime/malloc.go:795 +0x95 fp=0x7fdc669ade80 sp=0x7fdc669ade40
runtime.mallocgc.func1()
	/usr/lib/golang/src/runtime/malloc.go:690 +0x40 fp=0x7fdc669adeb0 sp=0x7fdc669ade80
runtime.systemstack(0xc420024000)
	/usr/lib/golang/src/runtime/asm_amd64.s:327 +0x72 fp=0x7fdc669adeb8 sp=0x7fdc669adeb0
runtime.mstart()
	/usr/lib/golang/src/runtime/proc.go:1132 fp=0x7fdc669adec0 sp=0x7fdc669adeb8

goroutine 497133 [running]:
runtime.systemstack_switch()
	/usr/lib/golang/src/runtime/asm_amd64.s:281 fp=0xc439371908 sp=0xc439371900
runtime.mallocgc(0x19afdfe3, 0x56208e6c9e20, 0xf343a01, 0x56208e1bf4e6)
	/usr/lib/golang/src/runtime/malloc.go:691 +0x942 fp=0xc4393719a8 sp=0xc439371908
runtime.makeslice(0x56208e6c9e20, 0x19afdfe3, 0x19afdfe3, 0x0, 0xc530e7e000, 0x19afdfd8)
	/usr/lib/golang/src/runtime/slice.go:54 +0x7d fp=0xc4393719f8 sp=0xc4393719a8
github.com/coreos/etcd/snap/snappb.(*Snapshot).Marshal(0xc439371bb8, 0xc4200ac400, 0xc530e7e000, 0x19afdfd8, 0x19afdfd8, 0xc40f343ad2)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/snap/snappb/snap.pb.go:53 +0x50 fp=0xc439371a50 sp=0xc4393719f8
github.com/coreos/etcd/snap.(*Snapshotter).save(0xc4201e6fb0, 0xc4211ac3f0, 0xc439371cb0, 0x56208e1ea300)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/snap/snapshotter.go:80 +0x26a fp=0xc439371c00 sp=0xc439371a50
github.com/coreos/etcd/snap.(*Snapshotter).SaveSnap(0xc4201e6fb0, 0xc49bec4000, 0x19afdfa8, 0x250ae124, 0xc4201d4500, 0x3, 0x3, 0x0, 0x0, 0x0, ...)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/snap/snapshotter.go:70 +0xef fp=0xc439371cc0 sp=0xc439371c00
github.com/coreos/etcd/etcdserver.(*storage).SaveSnap(0xc4201e7da0, 0xc49bec4000, 0x19afdfa8, 0x250ae124, 0xc4201d4500, 0x3, 0x3, 0x0, 0x0, 0x0, ...)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/etcdserver/storage.go:62 +0x123 fp=0xc439371d98 sp=0xc439371cc0
github.com/coreos/etcd/etcdserver.(*EtcdServer).snapshot.func1()
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/etcdserver/server.go:1422 +0x296 fp=0xc439371fa8 sp=0xc439371d98
github.com/coreos/etcd/etcdserver.(*EtcdServer).goAttach.func1(0xc42006b8c0, 0xc479ec9320)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/etcdserver/server.go:1638 +0x57 fp=0xc439371fd0 sp=0xc439371fa8
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc439371fd8 sp=0xc439371fd0
created by github.com/coreos/etcd/etcdserver.(*EtcdServer).goAttach
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/etcdserver/server.go:1639 +0x1aa

goroutine 1 [select, 383 minutes]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc420e8cbf8 sp=0xc420e8cbc8
runtime.selectgoImpl(0xc420e8d3a8, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc420e8ce70 sp=0xc420e8cbf8
runtime.selectgo(0xc420e8d3a8)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc420e8ce98 sp=0xc420e8ce70
github.com/coreos/etcd/etcdmain.startEtcdOrProxyV2()
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/etcdmain/etcd.go:171 +0xcd4 fp=0xc420e8df08 sp=0xc420e8ce98
github.com/coreos/etcd/etcdmain.Main()
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/etcdmain/main.go:39 +0x63 fp=0xc420e8df78 sp=0xc420e8df08
main.main()
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/cmd/etcd/main.go:28 +0x22 fp=0xc420e8df88 sp=0xc420e8df78
runtime.main()
	/usr/lib/golang/src/runtime/proc.go:185 +0x212 fp=0xc420e8dfe0 sp=0xc420e8df88
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420e8dfe8 sp=0xc420e8dfe0

goroutine 17 [syscall, 383 minutes, locked to thread]:
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420040fe8 sp=0xc420040fe0

goroutine 2 [force gc (idle), 3 minutes]:
runtime.gopark(0x56208e8625b8, 0x56208ec6a8d0, 0x56208e316cd2, 0xf, 0x14faac487114, 0x1)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc42002e768 sp=0xc42002e738
runtime.goparkunlock(0x56208ec6a8d0, 0x56208e316cd2, 0xf, 0xc420000114, 0x1)
	/usr/lib/golang/src/runtime/proc.go:277 +0x60 fp=0xc42002e7a8 sp=0xc42002e768
runtime.forcegchelper()
	/usr/lib/golang/src/runtime/proc.go:226 +0xa2 fp=0xc42002e7e0 sp=0xc42002e7a8
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc42002e7e8 sp=0xc42002e7e0
created by runtime.init.5
	/usr/lib/golang/src/runtime/proc.go:215 +0x37

goroutine 3 [GC sweep wait]:
runtime.gopark(0x56208e8625b8, 0x56208ec6ab80, 0x56208e314aad, 0xd, 0x56208db01914, 0x1)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc42002ef58 sp=0xc42002ef28
runtime.goparkunlock(0x56208ec6ab80, 0x56208e314aad, 0xd, 0x14, 0x1)
	/usr/lib/golang/src/runtime/proc.go:277 +0x60 fp=0xc42002ef98 sp=0xc42002ef58
runtime.bgsweep(0xc42001e070)
	/usr/lib/golang/src/runtime/mgcsweep.go:72 +0x132 fp=0xc42002efd8 sp=0xc42002ef98
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc42002efe0 sp=0xc42002efd8
created by runtime.gcenable
	/usr/lib/golang/src/runtime/mgc.go:212 +0x63

goroutine 18 [finalizer wait, 383 minutes]:
runtime.gopark(0x56208e8625b8, 0x56208ec8bd88, 0x56208e315c01, 0xe, 0x14, 0x1)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc42002a718 sp=0xc42002a6e8
runtime.goparkunlock(0x56208ec8bd88, 0x56208e315c01, 0xe, 0x14, 0x1)
	/usr/lib/golang/src/runtime/proc.go:277 +0x60 fp=0xc42002a758 sp=0xc42002a718
runtime.runfinq()
	/usr/lib/golang/src/runtime/mfinal.go:161 +0xb6 fp=0xc42002a7e0 sp=0xc42002a758
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc42002a7e8 sp=0xc42002a7e0
created by runtime.createfing
	/usr/lib/golang/src/runtime/mfinal.go:142 +0x64

goroutine 34 [chan receive]:
runtime.gopark(0x56208e8625b8, 0xc420020b38, 0x56208e3141f2, 0xc, 0xc420024d17, 0x3)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc42002ad58 sp=0xc42002ad28
runtime.goparkunlock(0xc420020b38, 0x56208e3141f2, 0xc, 0x56208db00b17, 0x3)
	/usr/lib/golang/src/runtime/proc.go:277 +0x60 fp=0xc42002ad98 sp=0xc42002ad58
runtime.chanrecv(0x56208e6c1420, 0xc420020ae0, 0xc42002af38, 0xc42002af01, 0x2002af70)
	/usr/lib/golang/src/runtime/chan.go:513 +0x375 fp=0xc42002ae38 sp=0xc42002ad98
runtime.chanrecv2(0x56208e6c1420, 0xc420020ae0, 0xc42002af38, 0xc420066801)
	/usr/lib/golang/src/runtime/chan.go:400 +0x35 fp=0xc42002ae70 sp=0xc42002ae38
github.com/coreos/etcd/pkg/logutil.(*MergeLogger).outputLoop(0xc4200d3b40)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/pkg/logutil/merge_logger.go:174 +0x96 fp=0xc42002afd8 sp=0xc42002ae70
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc42002afe0 sp=0xc42002afd8
created by github.com/coreos/etcd/pkg/logutil.NewMergeLogger
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/pkg/logutil/merge_logger.go:92 +0xd6

goroutine 47 [chan receive]:
runtime.gopark(0x56208e8625b8, 0xc420020b98, 0x56208e3141f2, 0xc, 0xc420024d17, 0x3)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc420d6ad58 sp=0xc420d6ad28
runtime.goparkunlock(0xc420020b98, 0x56208e3141f2, 0xc, 0x56208db00b17, 0x3)
	/usr/lib/golang/src/runtime/proc.go:277 +0x60 fp=0xc420d6ad98 sp=0xc420d6ad58
runtime.chanrecv(0x56208e6c1420, 0xc420020b40, 0xc420d6af38, 0xc420d6af01, 0x20d6af70)
	/usr/lib/golang/src/runtime/chan.go:513 +0x375 fp=0xc420d6ae38 sp=0xc420d6ad98
runtime.chanrecv2(0x56208e6c1420, 0xc420020b40, 0xc420d6af38, 0x1)
	/usr/lib/golang/src/runtime/chan.go:400 +0x35 fp=0xc420d6ae70 sp=0xc420d6ae38
github.com/coreos/etcd/pkg/logutil.(*MergeLogger).outputLoop(0xc4201988c0)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/pkg/logutil/merge_logger.go:174 +0x96 fp=0xc420d6afd8 sp=0xc420d6ae70
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420d6afe0 sp=0xc420d6afd8
created by github.com/coreos/etcd/pkg/logutil.NewMergeLogger
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/pkg/logutil/merge_logger.go:92 +0xd6

goroutine 14 [runnable]:
runtime.notetsleepg(0x56208ec6adf8, 0x1220ba, 0x0)
	/usr/lib/golang/src/runtime/lock_futex.go:205 +0x46 fp=0xc42002f760 sp=0xc42002f730
runtime.timerproc()
	/usr/lib/golang/src/runtime/time.go:209 +0x32b fp=0xc42002f7e0 sp=0xc42002f760
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc42002f7e8 sp=0xc42002f7e0
created by runtime.addtimerLocked
	/usr/lib/golang/src/runtime/time.go:116 +0xef

goroutine 71 [select]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc420344840 sp=0xc420344810
runtime.selectgoImpl(0xc420344dc8, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc420344ab8 sp=0xc420344840
runtime.selectgo(0xc420344dc8)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc420344ae0 sp=0xc420344ab8
github.com/coreos/etcd/rafthttp.(*streamWriter).run(0xc4200f1730)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/stream.go:149 +0x1716 fp=0xc420344fd8 sp=0xc420344ae0
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420344fe0 sp=0xc420344fd8
created by github.com/coreos/etcd/rafthttp.startStreamWriter
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/stream.go:130 +0x1ae

goroutine 69 [syscall, 383 minutes]:
runtime.notetsleepg(0x56208ec8c2a0, 0xffffffffffffffff, 0xc420175960)
	/usr/lib/golang/src/runtime/lock_futex.go:205 +0x46 fp=0xc42002c780 sp=0xc42002c750
os/signal.signal_recv(0x56208dadeeea)
	/usr/lib/golang/src/runtime/sigqueue.go:116 +0x106 fp=0xc42002c7a8 sp=0xc42002c780
os/signal.loop()
	/usr/lib/golang/src/os/signal/signal_unix.go:22 +0x24 fp=0xc42002c7e0 sp=0xc42002c7a8
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc42002c7e8 sp=0xc42002c7e0
created by os/signal.init.1
	/usr/lib/golang/src/os/signal/signal_unix.go:28 +0x43

goroutine 65 [runnable]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc420041c70 sp=0xc420041c40
runtime.selectgoImpl(0xc420041f48, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc420041ee8 sp=0xc420041c70
runtime.selectgo(0xc420041f48)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc420041f10 sp=0xc420041ee8
github.com/coreos/etcd/mvcc/backend.(*backend).run(0xc4201fa360)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/mvcc/backend/backend.go:193 +0x1c4 fp=0xc420041fd8 sp=0xc420041f10
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420041fe0 sp=0xc420041fd8
created by github.com/coreos/etcd/mvcc/backend.newBackend
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/mvcc/backend/backend.go:119 +0x26c

goroutine 87 [select, 86 minutes]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc420042c58 sp=0xc420042c28
runtime.selectgoImpl(0xc420042f48, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc420042ed0 sp=0xc420042c58
runtime.selectgo(0xc420042f48)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc420042ef8 sp=0xc420042ed0
github.com/coreos/etcd/wal.(*filePipeline).run(0xc4201f2bc0)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/wal/file_pipeline.go:89 +0x199 fp=0xc420042fd8 sp=0xc420042ef8
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420042fe0 sp=0xc420042fd8
created by github.com/coreos/etcd/wal.newFilePipeline
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/wal/file_pipeline.go:47 +0x136

goroutine 88 [select]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc420e88c58 sp=0xc420e88c28
runtime.selectgoImpl(0xc420e89e08, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc420e88ed0 sp=0xc420e88c58
runtime.selectgo(0xc420e89e08)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc420e88ef8 sp=0xc420e88ed0
github.com/coreos/etcd/raft.(*node).run(0xc4201fb080, 0xc4202081e0)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/raft/node.go:309 +0x11c0 fp=0xc420e89fd0 sp=0xc420e88ef8
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420e89fd8 sp=0xc420e89fd0
created by github.com/coreos/etcd/raft.StartNode
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/raft/node.go:206 +0x6b5

goroutine 89 [select]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc420348b88 sp=0xc420348b58
runtime.selectgoImpl(0xc420348e90, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc420348e00 sp=0xc420348b88
runtime.selectgo(0xc420348e90)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc420348e28 sp=0xc420348e00
github.com/coreos/etcd/lease.(*lessor).runLoop(0xc4201fb860)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/lease/lessor.go:447 +0x196 fp=0xc420348fd8 sp=0xc420348e28
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420348fe0 sp=0xc420348fd8
created by github.com/coreos/etcd/lease.newLessor
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/lease/lessor.go:169 +0x1e9

goroutine 90 [select, 383 minutes]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc420349c50 sp=0xc420349c20
runtime.selectgoImpl(0xc420349f48, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc420349ec8 sp=0xc420349c50
runtime.selectgo(0xc420349f48)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc420349ef0 sp=0xc420349ec8
github.com/coreos/etcd/pkg/schedule.(*fifo).run(0xc4201fb980)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/pkg/schedule/schedule.go:146 +0x35e fp=0xc420349fd8 sp=0xc420349ef0
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420349fe0 sp=0xc420349fd8
created by github.com/coreos/etcd/pkg/schedule.NewFIFOScheduler
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/pkg/schedule/schedule.go:71 +0x1ae

goroutine 91 [GC worker (idle)]:
runtime.gopark(0x56208e862438, 0xc420342060, 0x56208e317ce5, 0x10, 0x14, 0x0)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc42033b750 sp=0xc42033b720
runtime.gcBgMarkWorker(0xc420024000)
	/usr/lib/golang/src/runtime/mgc.go:1491 +0x13e fp=0xc42033b7d8 sp=0xc42033b750
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc42033b7e0 sp=0xc42033b7d8
created by runtime.gcBgMarkStartWorkers
	/usr/lib/golang/src/runtime/mgc.go:1412 +0x9a

goroutine 92 [GC worker (idle)]:
runtime.gopark(0x56208e862438, 0xc420342070, 0x56208e317ce5, 0x10, 0x14, 0x0)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc42033bf50 sp=0xc42033bf20
runtime.gcBgMarkWorker(0xc420025300)
	/usr/lib/golang/src/runtime/mgc.go:1491 +0x13e fp=0xc42033bfd8 sp=0xc42033bf50
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc42033bfe0 sp=0xc42033bfd8
created by runtime.gcBgMarkStartWorkers
	/usr/lib/golang/src/runtime/mgc.go:1412 +0x9a

goroutine 93 [runnable]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc42034ac50 sp=0xc42034ac20
runtime.selectgoImpl(0xc42034af48, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc42034aec8 sp=0xc42034ac50
runtime.selectgo(0xc42034af48)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc42034aef0 sp=0xc42034aec8
github.com/coreos/etcd/mvcc.(*watchableStore).syncWatchersLoop(0xc4200f54d0)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/mvcc/watchable_store.go:281 +0x203 fp=0xc42034afd8 sp=0xc42034aef0
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc42034afe0 sp=0xc42034afd8
created by github.com/coreos/etcd/mvcc.newWatchableStore
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/mvcc/watchable_store.go:84 +0x2ef

goroutine 94 [select, 383 minutes]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc42003dc30 sp=0xc42003dc00
runtime.selectgoImpl(0xc42003df10, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc42003dea8 sp=0xc42003dc30
runtime.selectgo(0xc42003df10)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc42003ded0 sp=0xc42003dea8
github.com/coreos/etcd/mvcc.(*watchableStore).syncVictimsLoop(0xc4200f54d0)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/mvcc/watchable_store.go:307 +0x1ce fp=0xc42003dfd8 sp=0xc42003ded0
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc42003dfe0 sp=0xc42003dfd8
created by github.com/coreos/etcd/mvcc.newWatchableStore
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/mvcc/watchable_store.go:85 +0x311

goroutine 95 [select, 383 minutes]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc42034b8a8 sp=0xc42034b878
runtime.selectgoImpl(0xc42034bc00, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc42034bb20 sp=0xc42034b8a8
runtime.selectgo(0xc42034bc00)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc42034bb48 sp=0xc42034bb20
github.com/coreos/etcd/rafthttp.(*pipeline).handle(0xc420066480)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/pipeline.go:80 +0x677 fp=0xc42034bfd8 sp=0xc42034bb48
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc42034bfe0 sp=0xc42034bfd8
created by github.com/coreos/etcd/rafthttp.(*pipeline).start
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/pipeline.go:65 +0xdc

goroutine 96 [select, 383 minutes]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc4203478a8 sp=0xc420347878
runtime.selectgoImpl(0xc420347c00, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc420347b20 sp=0xc4203478a8
runtime.selectgo(0xc420347c00)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc420347b48 sp=0xc420347b20
github.com/coreos/etcd/rafthttp.(*pipeline).handle(0xc420066480)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/pipeline.go:80 +0x677 fp=0xc420347fd8 sp=0xc420347b48
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420347fe0 sp=0xc420347fd8
created by github.com/coreos/etcd/rafthttp.(*pipeline).start
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/pipeline.go:65 +0xdc

goroutine 97 [select, 383 minutes]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc42003e8a8 sp=0xc42003e878
runtime.selectgoImpl(0xc42003ec00, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc42003eb20 sp=0xc42003e8a8
runtime.selectgo(0xc42003ec00)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc42003eb48 sp=0xc42003eb20
github.com/coreos/etcd/rafthttp.(*pipeline).handle(0xc420066480)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/pipeline.go:80 +0x677 fp=0xc42003efd8 sp=0xc42003eb48
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc42003efe0 sp=0xc42003efd8
created by github.com/coreos/etcd/rafthttp.(*pipeline).start
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/pipeline.go:65 +0xdc

goroutine 98 [select, 383 minutes]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc42003f8a8 sp=0xc42003f878
runtime.selectgoImpl(0xc42003fc00, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc42003fb20 sp=0xc42003f8a8
runtime.selectgo(0xc42003fc00)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc42003fb48 sp=0xc42003fb20
github.com/coreos/etcd/rafthttp.(*pipeline).handle(0xc420066480)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/pipeline.go:80 +0x677 fp=0xc42003ffd8 sp=0xc42003fb48
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc42003ffe0 sp=0xc42003ffd8
created by github.com/coreos/etcd/rafthttp.(*pipeline).start
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/pipeline.go:65 +0xdc

goroutine 99 [select]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc420113840 sp=0xc420113810
runtime.selectgoImpl(0xc420113dc8, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc420113ab8 sp=0xc420113840
runtime.selectgo(0xc420113dc8)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc420113ae0 sp=0xc420113ab8
github.com/coreos/etcd/rafthttp.(*streamWriter).run(0xc42001e230)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/stream.go:149 +0x1716 fp=0xc420113fd8 sp=0xc420113ae0
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420113fe0 sp=0xc420113fd8
created by github.com/coreos/etcd/rafthttp.startStreamWriter
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/stream.go:130 +0x1ae

goroutine 72 [select]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc420d718f8 sp=0xc420d718c8
runtime.selectgoImpl(0xc420d71cf8, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc420d71b70 sp=0xc420d718f8
runtime.selectgo(0xc420d71cf8)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc420d71b98 sp=0xc420d71b70
github.com/coreos/etcd/rafthttp.startPeer.func1(0xc4200f4090, 0x56208ec3dca0, 0xc42006b8c0, 0x7fdc68b2a3e8, 0xc420016100)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/peer.go:156 +0x273 fp=0xc420d71fb8 sp=0xc420d71b98
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420d71fc0 sp=0xc420d71fb8
created by github.com/coreos/etcd/rafthttp.startPeer
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/peer.go:165 +0x75e

goroutine 73 [select, 107 minutes]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc4201118f8 sp=0xc4201118c8
runtime.selectgoImpl(0xc420111cf8, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc420111b70 sp=0xc4201118f8
runtime.selectgo(0xc420111cf8)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc420111b98 sp=0xc420111b70
github.com/coreos/etcd/rafthttp.startPeer.func2(0xc4200f4090, 0x56208ec3dca0, 0xc42006b8c0, 0x7fdc68b2a3e8, 0xc420016100)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/peer.go:172 +0x273 fp=0xc420111fb8 sp=0xc420111b98
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420111fc0 sp=0xc420111fb8
created by github.com/coreos/etcd/rafthttp.startPeer
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/peer.go:181 +0x7b1

goroutine 74 [IO wait]:
runtime.gopark(0x56208e862580, 0x7fdc68ae2c28, 0x56208e30f82a, 0x7, 0x56208db23d1b, 0x5)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc4200570d8 sp=0xc4200570a8
runtime.netpollblock(0x7fdc68ae2c00, 0x72, 0xb)
	/usr/lib/golang/src/runtime/netpoll.go:345 +0xc9 fp=0xc420057120 sp=0xc4200570d8
net.runtime_pollWait(0x7fdc68ae2c00, 0x72, 0xb)
	/usr/lib/golang/src/runtime/netpoll.go:164 +0x5b fp=0xc420057148 sp=0xc420057120
net.(*pollDesc).wait(0xc42001e378, 0x72, 0x56208ec35020, 0x56208ec2ebc0)
	/usr/lib/golang/src/net/fd_poll_runtime.go:75 +0x3a fp=0xc420057170 sp=0xc420057148
net.(*pollDesc).waitRead(0xc42001e378, 0xc4201f6000, 0x1000)
	/usr/lib/golang/src/net/fd_poll_runtime.go:80 +0x36 fp=0xc4200571a0 sp=0xc420057170
net.(*netFD).Read(0xc42001e310, 0xc4201f6000, 0x1000, 0x1000, 0x0, 0x56208ec35020, 0x56208ec2ebc0)
	/usr/lib/golang/src/net/fd_unix.go:250 +0x1b9 fp=0xc420057208 sp=0xc4200571a0
net.(*conn).Read(0xc42007c040, 0xc4201f6000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/lib/golang/src/net/net.go:181 +0x72 fp=0xc420057278 sp=0xc420057208
github.com/coreos/etcd/pkg/transport.timeoutConn.Read(0x56208ec41ea0, 0xc42007c040, 0x12a05f200, 0x12a05f200, 0xc4201f6000, 0x1000, 0x1000, 0x6, 0x0, 0x0)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/pkg/transport/timeout_conn.go:43 +0x5f fp=0xc4200572c0 sp=0xc420057278
github.com/coreos/etcd/pkg/transport.(*timeoutConn).Read(0xc4200d2b80, 0xc4201f6000, 0x1000, 0x1000, 0x6, 0x0, 0x0)
	<autogenerated>:42 +0x88 fp=0xc420057320 sp=0xc4200572c0
net/http.(*persistConn).Read(0xc420090480, 0xc4201f6000, 0x1000, 0x1000, 0x6, 0x0, 0x56208dc1dda4)
	/usr/lib/golang/src/net/http/transport.go:1316 +0x14d fp=0xc420057398 sp=0xc420057320
bufio.(*Reader).fill(0xc420020420)
	/usr/lib/golang/src/bufio/bufio.go:97 +0x119 fp=0xc4200573f8 sp=0xc420057398
bufio.(*Reader).ReadSlice(0xc420020420, 0xc42002040a, 0xc42006dfd0, 0x2, 0x2, 0x2, 0x0)
	/usr/lib/golang/src/bufio/bufio.go:338 +0xbd fp=0xc420057458 sp=0xc4200573f8
net/http/internal.readChunkLine(0xc420020420, 0x56208ec305e0, 0xc420020420, 0xc42006dfd0, 0x2, 0x2)
	/usr/lib/golang/src/net/http/internal/chunked.go:122 +0x36 fp=0xc4200574b8 sp=0xc420057458
net/http/internal.(*chunkedReader).beginChunk(0xc42006dfb0)
	/usr/lib/golang/src/net/http/internal/chunked.go:48 +0x34 fp=0xc420057518 sp=0xc4200574b8
net/http/internal.(*chunkedReader).Read(0xc42006dfb0, 0xc420069938, 0x1, 0x1, 0xc4791c1c00, 0x0, 0x56208dd3166f)
	/usr/lib/golang/src/net/http/internal/chunked.go:93 +0x12a fp=0xc420057598 sp=0xc420057518
net/http.(*body).readLocked(0xc4201ae600, 0xc420069938, 0x1, 0x1, 0xc42006f380, 0x0, 0x0)
	/usr/lib/golang/src/net/http/transfer.go:761 +0x63 fp=0xc4200575f8 sp=0xc420057598
net/http.(*body).Read(0xc4201ae600, 0xc420069938, 0x1, 0x1, 0x0, 0x0, 0x0)
	/usr/lib/golang/src/net/http/transfer.go:753 +0xff fp=0xc420057648 sp=0xc4200575f8
net/http.(*bodyEOFSignal).Read(0xc4201ae640, 0xc420069938, 0x1, 0x1, 0x0, 0x0, 0x0)
	/usr/lib/golang/src/net/http/transport.go:2035 +0xeb fp=0xc4200576b0 sp=0xc420057648
io.ReadAtLeast(0x56208ec32ae0, 0xc4201ae640, 0xc420069938, 0x1, 0x1, 0x1, 0x56208e834f20, 0x56208e123001, 0xc4366a7440)
	/usr/lib/golang/src/io/io.go:307 +0xab fp=0xc420057710 sp=0xc4200576b0
io.ReadFull(0x56208ec32ae0, 0xc4201ae640, 0xc420069938, 0x1, 0x1, 0x0, 0xc420182240, 0xc420057868)
	/usr/lib/golang/src/io/io.go:325 +0x5a fp=0xc420057768 sp=0xc420057710
github.com/coreos/etcd/rafthttp.(*msgAppV2Decoder).decode(0xc42001c600, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/msgappv2_codec.go:179 +0xb7 fp=0xc420057950 sp=0xc420057768
github.com/coreos/etcd/rafthttp.(*streamReader).decodeLoop(0xc42007a180, 0x56208ec37020, 0xc4201ae640, 0x56208e3111c7, 0x8, 0x2, 0x0)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/stream.go:358 +0x293 fp=0xc420057df8 sp=0xc420057950
github.com/coreos/etcd/rafthttp.(*streamReader).run(0xc42007a180)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/stream.go:311 +0x69f fp=0xc420057fd8 sp=0xc420057df8
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420057fe0 sp=0xc420057fd8
created by github.com/coreos/etcd/rafthttp.(*streamReader).start
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/stream.go:296 +0xb9

goroutine 75 [runnable]:
runtime.gopark(0x56208e862580, 0x7fdc68ae2b68, 0x56208e30f82a, 0x7, 0x56208db23d1b, 0x5)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc4209f0fa0 sp=0xc4209f0f70
runtime.netpollblock(0x7fdc68ae2b40, 0x72, 0xb)
	/usr/lib/golang/src/runtime/netpoll.go:345 +0xc9 fp=0xc4209f0fe8 sp=0xc4209f0fa0
net.runtime_pollWait(0x7fdc68ae2b40, 0x72, 0xd)
	/usr/lib/golang/src/runtime/netpoll.go:164 +0x5b fp=0xc4209f1010 sp=0xc4209f0fe8
net.(*pollDesc).wait(0xc42001e4c8, 0x72, 0x56208ec35020, 0x56208ec2ebc0)
	/usr/lib/golang/src/net/fd_poll_runtime.go:75 +0x3a fp=0xc4209f1038 sp=0xc4209f1010
net.(*pollDesc).waitRead(0xc42001e4c8, 0xc4201f8000, 0x1000)
	/usr/lib/golang/src/net/fd_poll_runtime.go:80 +0x36 fp=0xc4209f1068 sp=0xc4209f1038
net.(*netFD).Read(0xc42001e460, 0xc4201f8000, 0x1000, 0x1000, 0x0, 0x56208ec35020, 0x56208ec2ebc0)
	/usr/lib/golang/src/net/fd_unix.go:250 +0x1b9 fp=0xc4209f10d0 sp=0xc4209f1068
net.(*conn).Read(0xc42007c048, 0xc4201f8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/lib/golang/src/net/net.go:181 +0x72 fp=0xc4209f1140 sp=0xc4209f10d0
github.com/coreos/etcd/pkg/transport.timeoutConn.Read(0x56208ec41ea0, 0xc42007c048, 0x12a05f200, 0x12a05f200, 0xc4201f8000, 0x1000, 0x1000, 0x2a, 0x0, 0x0)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/pkg/transport/timeout_conn.go:43 +0x5f fp=0xc4209f1188 sp=0xc4209f1140
github.com/coreos/etcd/pkg/transport.(*timeoutConn).Read(0xc4200d2cc0, 0xc4201f8000, 0x1000, 0x1000, 0x2a, 0x0, 0x0)
	<autogenerated>:42 +0x88 fp=0xc4209f11e8 sp=0xc4209f1188
net/http.(*persistConn).Read(0xc4200906c0, 0xc4201f8000, 0x1000, 0x1000, 0x2a, 0x0, 0x0)
	/usr/lib/golang/src/net/http/transport.go:1316 +0x14d fp=0xc4209f1260 sp=0xc4209f11e8
bufio.(*Reader).fill(0xc420020840)
	/usr/lib/golang/src/bufio/bufio.go:97 +0x119 fp=0xc4209f12c0 sp=0xc4209f1260
bufio.(*Reader).ReadSlice(0xc420020840, 0xa, 0x1000, 0xc4201f8000, 0x2, 0x56208dc1dda4, 0xc4784af1f8)
	/usr/lib/golang/src/bufio/bufio.go:338 +0xbd fp=0xc4209f1320 sp=0xc4209f12c0
net/http/internal.readChunkLine(0xc420020840, 0xc4209f1890, 0xc420a516d4, 0x0, 0x8, 0x0)
	/usr/lib/golang/src/net/http/internal/chunked.go:122 +0x36 fp=0xc4209f1380 sp=0xc4209f1320
net/http/internal.(*chunkedReader).beginChunk(0xc42001b170)
	/usr/lib/golang/src/net/http/internal/chunked.go:48 +0x34 fp=0xc4209f13e0 sp=0xc4209f1380
net/http/internal.(*chunkedReader).Read(0xc42001b170, 0xc4784af278, 0x8, 0x8, 0xc4759077c0, 0x56208e18545a, 0x56208dd3166f)
	/usr/lib/golang/src/net/http/internal/chunked.go:93 +0x12a fp=0xc4209f1460 sp=0xc4209f13e0
net/http.(*body).readLocked(0xc420016980, 0xc4784af278, 0x8, 0x8, 0x8, 0x8, 0x0)
	/usr/lib/golang/src/net/http/transfer.go:761 +0x63 fp=0xc4209f14c0 sp=0xc4209f1460
net/http.(*body).Read(0xc420016980, 0xc4784af278, 0x8, 0x8, 0x0, 0x0, 0x0)
	/usr/lib/golang/src/net/http/transfer.go:753 +0xff fp=0xc4209f1510 sp=0xc4209f14c0
net/http.(*bodyEOFSignal).Read(0xc420016c40, 0xc4784af278, 0x8, 0x8, 0x0, 0x0, 0x0)
	/usr/lib/golang/src/net/http/transport.go:2035 +0xeb fp=0xc4209f1578 sp=0xc4209f1510
io.ReadAtLeast(0x56208ec32ae0, 0xc420016c40, 0xc4784af278, 0x8, 0x8, 0x8, 0x56208e6d8de0, 0x1, 0xc4784af278)
	/usr/lib/golang/src/io/io.go:307 +0xab fp=0xc4209f15d8 sp=0xc4209f1578
io.ReadFull(0x56208ec32ae0, 0xc420016c40, 0xc4784af278, 0x8, 0x8, 0xc420069939, 0x1, 0xc4784af249)
	/usr/lib/golang/src/io/io.go:325 +0x5a fp=0xc4209f1630 sp=0xc4209f15d8
encoding/binary.Read(0x56208ec32ae0, 0xc420016c40, 0x56208ec418a0, 0x56208ec8be78, 0x56208e6b45c0, 0xc4784af270, 0x56208db15f01, 0xc4209f1828)
	/usr/lib/golang/src/encoding/binary/binary.go:170 +0xec fp=0xc4209f17b8 sp=0xc4209f1630
github.com/coreos/etcd/rafthttp.(*messageDecoder).decode(0xc420018eb0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/msg_codec.go:53 +0x11f fp=0xc4209f1950 sp=0xc4209f17b8
github.com/coreos/etcd/rafthttp.(*streamReader).decodeLoop(0xc42007a280, 0x56208ec37020, 0xc420016c40, 0x56208e310210, 0x7, 0x2, 0x0)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/stream.go:358 +0x293 fp=0xc4209f1df8 sp=0xc4209f1950
github.com/coreos/etcd/rafthttp.(*streamReader).run(0xc42007a280)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/stream.go:311 +0x69f fp=0xc4209f1fd8 sp=0xc4209f1df8
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc4209f1fe0 sp=0xc4209f1fd8
created by github.com/coreos/etcd/rafthttp.(*streamReader).start
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/stream.go:296 +0xb9

goroutine 76 [select]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc420d6fbd0 sp=0xc420d6fba0
runtime.selectgoImpl(0xc420d6ff20, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc420d6fe48 sp=0xc420d6fbd0
runtime.selectgo(0xc420d6ff20)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc420d6fe70 sp=0xc420d6fe48
github.com/xiang90/probing.(*prober).AddHTTP.func1(0xc420016680, 0xc4200182b0, 0x1, 0x1, 0xc42000c520, 0xc4201d2460)
	/usr/share/gocode/src/github.com/xiang90/probing/prober.go:55 +0x497 fp=0xc420d6ffb0 sp=0xc420d6fe70
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420d6ffb8 sp=0xc420d6ffb0
created by github.com/xiang90/probing.(*prober).AddHTTP
	/usr/share/gocode/src/github.com/xiang90/probing/prober.go:85 +0x1db

goroutine 77 [select]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc42010dba8 sp=0xc42010db78
runtime.selectgoImpl(0xc42010df30, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc42010de20 sp=0xc42010dba8
runtime.selectgo(0xc42010df30)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc42010de48 sp=0xc42010de20
github.com/coreos/etcd/rafthttp.monitorProbingStatus(0x56208ec416c0, 0xc4201d2460, 0xc420018270, 0x10)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/probing_status.go:51 +0x508 fp=0xc42010dfc0 sp=0xc42010de48
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc42010dfc8 sp=0xc42010dfc0
created by github.com/coreos/etcd/rafthttp.addPeerToProber
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/probing_status.go:43 +0x2d2

goroutine 78 [select, 383 minutes]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc4209f88a8 sp=0xc4209f8878
runtime.selectgoImpl(0xc4209f8c00, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc4209f8b20 sp=0xc4209f88a8
runtime.selectgo(0xc4209f8c00)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc4209f8b48 sp=0xc4209f8b20
github.com/coreos/etcd/rafthttp.(*pipeline).handle(0xc4200669c0)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/pipeline.go:80 +0x677 fp=0xc4209f8fd8 sp=0xc4209f8b48
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc4209f8fe0 sp=0xc4209f8fd8
created by github.com/coreos/etcd/rafthttp.(*pipeline).start
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/pipeline.go:65 +0xdc

goroutine 79 [select, 383 minutes]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc4209f98a8 sp=0xc4209f9878
runtime.selectgoImpl(0xc4209f9c00, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc4209f9b20 sp=0xc4209f98a8
runtime.selectgo(0xc4209f9c00)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc4209f9b48 sp=0xc4209f9b20
github.com/coreos/etcd/rafthttp.(*pipeline).handle(0xc4200669c0)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/pipeline.go:80 +0x677 fp=0xc4209f9fd8 sp=0xc4209f9b48
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc4209f9fe0 sp=0xc4209f9fd8
created by github.com/coreos/etcd/rafthttp.(*pipeline).start
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/pipeline.go:65 +0xdc

goroutine 80 [select, 383 minutes]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc4209f78a8 sp=0xc4209f7878
runtime.selectgoImpl(0xc4209f7c00, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc4209f7b20 sp=0xc4209f78a8
runtime.selectgo(0xc4209f7c00)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc4209f7b48 sp=0xc4209f7b20
github.com/coreos/etcd/rafthttp.(*pipeline).handle(0xc4200669c0)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/pipeline.go:80 +0x677 fp=0xc4209f7fd8 sp=0xc4209f7b48
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc4209f7fe0 sp=0xc4209f7fd8
created by github.com/coreos/etcd/rafthttp.(*pipeline).start
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/pipeline.go:65 +0xdc

goroutine 81 [select, 383 minutes]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc4206818a8 sp=0xc420681878
runtime.selectgoImpl(0xc420681c00, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc420681b20 sp=0xc4206818a8
runtime.selectgo(0xc420681c00)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc420681b48 sp=0xc420681b20
github.com/coreos/etcd/rafthttp.(*pipeline).handle(0xc4200669c0)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/pipeline.go:80 +0x677 fp=0xc420681fd8 sp=0xc420681b48
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420681fe0 sp=0xc420681fd8
created by github.com/coreos/etcd/rafthttp.(*pipeline).start
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/pipeline.go:65 +0xdc

goroutine 100 [select]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc42010f840 sp=0xc42010f810
runtime.selectgoImpl(0xc42010fdc8, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc42010fab8 sp=0xc42010f840
runtime.selectgo(0xc42010fdc8)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc42010fae0 sp=0xc42010fab8
github.com/coreos/etcd/rafthttp.(*streamWriter).run(0xc42001e070)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/stream.go:149 +0x1716 fp=0xc42010ffd8 sp=0xc42010fae0
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc42010ffe0 sp=0xc42010ffd8
created by github.com/coreos/etcd/rafthttp.startStreamWriter
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/stream.go:130 +0x1ae

goroutine 107 [select]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc42010c840 sp=0xc42010c810
runtime.selectgoImpl(0xc42010cdc8, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc42010cab8 sp=0xc42010c840
runtime.selectgo(0xc42010cdc8)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc42010cae0 sp=0xc42010cab8
github.com/coreos/etcd/rafthttp.(*streamWriter).run(0xc42001e4d0)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/stream.go:149 +0x1716 fp=0xc42010cfd8 sp=0xc42010cae0
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc42010cfe0 sp=0xc42010cfd8
created by github.com/coreos/etcd/rafthttp.startStreamWriter
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/stream.go:130 +0x1ae

goroutine 105 [select, 383 minutes]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc4203458f0 sp=0xc4203458c0
runtime.selectgoImpl(0xc420345ee0, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc420345b68 sp=0xc4203458f0
runtime.selectgo(0xc420345ee0)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc420345b90 sp=0xc420345b68
net/http.(*persistConn).readLoop(0xc420090480)
	/usr/lib/golang/src/net/http/transport.go:1599 +0x9ee fp=0xc420345fd8 sp=0xc420345b90
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420345fe0 sp=0xc420345fd8
created by net/http.(*Transport).dialConn
	/usr/lib/golang/src/net/http/transport.go:1117 +0xa37

goroutine 110 [select, 107 minutes]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc4210ed8f8 sp=0xc4210ed8c8
runtime.selectgoImpl(0xc4210edcf8, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc4210edb70 sp=0xc4210ed8f8
runtime.selectgo(0xc4210edcf8)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc4210edb98 sp=0xc4210edb70
github.com/coreos/etcd/rafthttp.startPeer.func1(0xc420022240, 0x56208ec3dca0, 0xc42006b8c0, 0x7fdc68b2a3e8, 0xc420070200)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/peer.go:156 +0x273 fp=0xc4210edfb8 sp=0xc4210edb98
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc4210edfc0 sp=0xc4210edfb8
created by github.com/coreos/etcd/rafthttp.startPeer
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/peer.go:165 +0x75e

goroutine 108 [select, 383 minutes]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc4203468f0 sp=0xc4203468c0
runtime.selectgoImpl(0xc420346ee0, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc420346b68 sp=0xc4203468f0
runtime.selectgo(0xc420346ee0)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc420346b90 sp=0xc420346b68
net/http.(*persistConn).readLoop(0xc4200906c0)
	/usr/lib/golang/src/net/http/transport.go:1599 +0x9ee fp=0xc420346fd8 sp=0xc420346b90
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420346fe0 sp=0xc420346fd8
created by net/http.(*Transport).dialConn
	/usr/lib/golang/src/net/http/transport.go:1117 +0xa37

goroutine 106 [select, 383 minutes]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc420110be8 sp=0xc420110bb8
runtime.selectgoImpl(0xc420110f48, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc420110e60 sp=0xc420110be8
runtime.selectgo(0xc420110f48)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc420110e88 sp=0xc420110e60
net/http.(*persistConn).writeLoop(0xc420090480)
	/usr/lib/golang/src/net/http/transport.go:1704 +0x43c fp=0xc420110fd8 sp=0xc420110e88
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420110fe0 sp=0xc420110fd8
created by net/http.(*Transport).dialConn
	/usr/lib/golang/src/net/http/transport.go:1118 +0xa5c

goroutine 109 [select, 383 minutes]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc4209f3be8 sp=0xc4209f3bb8
runtime.selectgoImpl(0xc4209f3f48, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc4209f3e60 sp=0xc4209f3be8
runtime.selectgo(0xc4209f3f48)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc4209f3e88 sp=0xc4209f3e60
net/http.(*persistConn).writeLoop(0xc4200906c0)
	/usr/lib/golang/src/net/http/transport.go:1704 +0x43c fp=0xc4209f3fd8 sp=0xc4209f3e88
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc4209f3fe0 sp=0xc4209f3fd8
created by net/http.(*Transport).dialConn
	/usr/lib/golang/src/net/http/transport.go:1118 +0xa5c

goroutine 111 [select, 107 minutes]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc4790838f8 sp=0xc4790838c8
runtime.selectgoImpl(0xc479083cf8, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc479083b70 sp=0xc4790838f8
runtime.selectgo(0xc479083cf8)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc479083b98 sp=0xc479083b70
github.com/coreos/etcd/rafthttp.startPeer.func2(0xc420022240, 0x56208ec3dca0, 0xc42006b8c0, 0x7fdc68b2a3e8, 0xc420070200)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/peer.go:172 +0x273 fp=0xc479083fb8 sp=0xc479083b98
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc479083fc0 sp=0xc479083fb8
created by github.com/coreos/etcd/rafthttp.startPeer
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/peer.go:181 +0x7b1

goroutine 112 [IO wait]:
runtime.gopark(0x56208e862580, 0x7fd9dee1d080, 0x56208e30f82a, 0x7, 0x56208db23d1b, 0x5)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc4200550d8 sp=0xc4200550a8
runtime.netpollblock(0x7fd9dee1d058, 0x72, 0xb)
	/usr/lib/golang/src/runtime/netpoll.go:345 +0xc9 fp=0xc420055120 sp=0xc4200550d8
net.runtime_pollWait(0x7fd9dee1d058, 0x72, 0xe)
	/usr/lib/golang/src/runtime/netpoll.go:164 +0x5b fp=0xc420055148 sp=0xc420055120
net.(*pollDesc).wait(0xc47d76f258, 0x72, 0x56208ec35020, 0x56208ec2ebc0)
	/usr/lib/golang/src/net/fd_poll_runtime.go:75 +0x3a fp=0xc420055170 sp=0xc420055148
net.(*pollDesc).waitRead(0xc47d76f258, 0xc4380cb000, 0x1000)
	/usr/lib/golang/src/net/fd_poll_runtime.go:80 +0x36 fp=0xc4200551a0 sp=0xc420055170
net.(*netFD).Read(0xc47d76f1f0, 0xc4380cb000, 0x1000, 0x1000, 0x0, 0x56208ec35020, 0x56208ec2ebc0)
	/usr/lib/golang/src/net/fd_unix.go:250 +0x1b9 fp=0xc420055208 sp=0xc4200551a0
net.(*conn).Read(0xc433448660, 0xc4380cb000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/lib/golang/src/net/net.go:181 +0x72 fp=0xc420055278 sp=0xc420055208
github.com/coreos/etcd/pkg/transport.timeoutConn.Read(0x56208ec41ea0, 0xc433448660, 0x12a05f200, 0x12a05f200, 0xc4380cb000, 0x1000, 0x1000, 0x6, 0x0, 0x0)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/pkg/transport/timeout_conn.go:43 +0x5f fp=0xc4200552c0 sp=0xc420055278
github.com/coreos/etcd/pkg/transport.(*timeoutConn).Read(0xc4381fcb20, 0xc4380cb000, 0x1000, 0x1000, 0x6, 0x0, 0x0)
	<autogenerated>:42 +0x88 fp=0xc420055320 sp=0xc4200552c0
net/http.(*persistConn).Read(0xc438094c60, 0xc4380cb000, 0x1000, 0x1000, 0x6, 0x0, 0x56208dc1dda4)
	/usr/lib/golang/src/net/http/transport.go:1316 +0x14d fp=0xc420055398 sp=0xc420055320
bufio.(*Reader).fill(0xc4381ff4a0)
	/usr/lib/golang/src/bufio/bufio.go:97 +0x119 fp=0xc4200553f8 sp=0xc420055398
bufio.(*Reader).ReadSlice(0xc4381ff4a0, 0xc4381ff40a, 0xc4381fad70, 0x2, 0x2, 0x2, 0x0)
	/usr/lib/golang/src/bufio/bufio.go:338 +0xbd fp=0xc420055458 sp=0xc4200553f8
net/http/internal.readChunkLine(0xc4381ff4a0, 0x56208ec305e0, 0xc4381ff4a0, 0xc4381fad70, 0x2, 0x2)
	/usr/lib/golang/src/net/http/internal/chunked.go:122 +0x36 fp=0xc4200554b8 sp=0xc420055458
net/http/internal.(*chunkedReader).beginChunk(0xc4381fad50)
	/usr/lib/golang/src/net/http/internal/chunked.go:48 +0x34 fp=0xc420055518 sp=0xc4200554b8
net/http/internal.(*chunkedReader).Read(0xc4381fad50, 0xc4380b5218, 0x1, 0x1, 0xc4886069c0, 0x0, 0x56208dd3166f)
	/usr/lib/golang/src/net/http/internal/chunked.go:93 +0x12a fp=0xc420055598 sp=0xc420055518
net/http.(*body).readLocked(0xc437e7d3c0, 0xc4380b5218, 0x1, 0x1, 0xc4201971e0, 0x0, 0x0)
	/usr/lib/golang/src/net/http/transfer.go:761 +0x63 fp=0xc4200555f8 sp=0xc420055598
net/http.(*body).Read(0xc437e7d3c0, 0xc4380b5218, 0x1, 0x1, 0x0, 0x0, 0x0)
	/usr/lib/golang/src/net/http/transfer.go:753 +0xff fp=0xc420055648 sp=0xc4200555f8
net/http.(*bodyEOFSignal).Read(0xc437e7d400, 0xc4380b5218, 0x1, 0x1, 0x0, 0x0, 0x0)
	/usr/lib/golang/src/net/http/transport.go:2035 +0xeb fp=0xc4200556b0 sp=0xc420055648
io.ReadAtLeast(0x56208ec32ae0, 0xc437e7d400, 0xc4380b5218, 0x1, 0x1, 0x1, 0x56208e834f20, 0x56208e123001, 0xc437a4dc20)
	/usr/lib/golang/src/io/io.go:307 +0xab fp=0xc420055710 sp=0xc4200556b0
io.ReadFull(0x56208ec32ae0, 0xc437e7d400, 0xc4380b5218, 0x1, 0x1, 0x0, 0xc420182240, 0xc420055868)
	/usr/lib/golang/src/io/io.go:325 +0x5a fp=0xc420055768 sp=0xc420055710
github.com/coreos/etcd/rafthttp.(*msgAppV2Decoder).decode(0xc4332fae80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/msgappv2_codec.go:179 +0xb7 fp=0xc420055950 sp=0xc420055768
github.com/coreos/etcd/rafthttp.(*streamReader).decodeLoop(0xc42001c280, 0x56208ec37020, 0xc437e7d400, 0x56208e3111c7, 0x8, 0x2, 0x0)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/stream.go:358 +0x293 fp=0xc420055df8 sp=0xc420055950
github.com/coreos/etcd/rafthttp.(*streamReader).run(0xc42001c280)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/stream.go:311 +0x69f fp=0xc420055fd8 sp=0xc420055df8
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420055fe0 sp=0xc420055fd8
created by github.com/coreos/etcd/rafthttp.(*streamReader).start
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/stream.go:296 +0xb9

goroutine 113 [IO wait]:
runtime.gopark(0x56208e862580, 0x7fdc647ad338, 0x56208e30f82a, 0x7, 0x56208db23d1b, 0x5)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc420d74fa0 sp=0xc420d74f70
runtime.netpollblock(0x7fdc647ad310, 0x72, 0xb)
	/usr/lib/golang/src/runtime/netpoll.go:345 +0xc9 fp=0xc420d74fe8 sp=0xc420d74fa0
net.runtime_pollWait(0x7fdc647ad310, 0x72, 0x14)
	/usr/lib/golang/src/runtime/netpoll.go:164 +0x5b fp=0xc420d75010 sp=0xc420d74fe8
net.(*pollDesc).wait(0xc4201c9fe8, 0x72, 0x56208ec35020, 0x56208ec2ebc0)
	/usr/lib/golang/src/net/fd_poll_runtime.go:75 +0x3a fp=0xc420d75038 sp=0xc420d75010
net.(*pollDesc).waitRead(0xc4201c9fe8, 0xc47e22c000, 0x1000)
	/usr/lib/golang/src/net/fd_poll_runtime.go:80 +0x36 fp=0xc420d75068 sp=0xc420d75038
net.(*netFD).Read(0xc4201c9f80, 0xc47e22c000, 0x1000, 0x1000, 0x0, 0x56208ec35020, 0x56208ec2ebc0)
	/usr/lib/golang/src/net/fd_unix.go:250 +0x1b9 fp=0xc420d750d0 sp=0xc420d75068
net.(*conn).Read(0xc488c067b0, 0xc47e22c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/lib/golang/src/net/net.go:181 +0x72 fp=0xc420d75140 sp=0xc420d750d0
github.com/coreos/etcd/pkg/transport.timeoutConn.Read(0x56208ec41ea0, 0xc488c067b0, 0x12a05f200, 0x12a05f200, 0xc47e22c000, 0x1000, 0x1000, 0x2a, 0x0, 0x0)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/pkg/transport/timeout_conn.go:43 +0x5f fp=0xc420d75188 sp=0xc420d75140
github.com/coreos/etcd/pkg/transport.(*timeoutConn).Read(0xc43bfa8e40, 0xc47e22c000, 0x1000, 0x1000, 0x2a, 0x0, 0x0)
	<autogenerated>:42 +0x88 fp=0xc420d751e8 sp=0xc420d75188
net/http.(*persistConn).Read(0xc488d47b00, 0xc47e22c000, 0x1000, 0x1000, 0x2a, 0x0, 0x0)
	/usr/lib/golang/src/net/http/transport.go:1316 +0x14d fp=0xc420d75260 sp=0xc420d751e8
bufio.(*Reader).fill(0xc47b8333e0)
	/usr/lib/golang/src/bufio/bufio.go:97 +0x119 fp=0xc420d752c0 sp=0xc420d75260
bufio.(*Reader).ReadSlice(0xc47b8333e0, 0xa, 0x1000, 0xc47e22c000, 0x2, 0x56208dc1dda4, 0xc4784ae2e0)
	/usr/lib/golang/src/bufio/bufio.go:338 +0xbd fp=0xc420d75320 sp=0xc420d752c0
net/http/internal.readChunkLine(0xc47b8333e0, 0xc420d75890, 0xc420a512f4, 0x0, 0x8, 0x0)
	/usr/lib/golang/src/net/http/internal/chunked.go:122 +0x36 fp=0xc420d75380 sp=0xc420d75320
net/http/internal.(*chunkedReader).beginChunk(0xc47500c960)
	/usr/lib/golang/src/net/http/internal/chunked.go:48 +0x34 fp=0xc420d753e0 sp=0xc420d75380
net/http/internal.(*chunkedReader).Read(0xc47500c960, 0xc4784aecd0, 0x8, 0x8, 0xc518c83a80, 0x56208e18545a, 0x56208dd3166f)
	/usr/lib/golang/src/net/http/internal/chunked.go:93 +0x12a fp=0xc420d75460 sp=0xc420d753e0
net/http.(*body).readLocked(0xc47a1690c0, 0xc4784aecd0, 0x8, 0x8, 0x8, 0x8, 0x0)
	/usr/lib/golang/src/net/http/transfer.go:761 +0x63 fp=0xc420d754c0 sp=0xc420d75460
net/http.(*body).Read(0xc47a1690c0, 0xc4784aecd0, 0x8, 0x8, 0x0, 0x0, 0x0)
	/usr/lib/golang/src/net/http/transfer.go:753 +0xff fp=0xc420d75510 sp=0xc420d754c0
net/http.(*bodyEOFSignal).Read(0xc47a169100, 0xc4784aecd0, 0x8, 0x8, 0x0, 0x0, 0x0)
	/usr/lib/golang/src/net/http/transport.go:2035 +0xeb fp=0xc420d75578 sp=0xc420d75510
io.ReadAtLeast(0x56208ec32ae0, 0xc47a169100, 0xc4784aecd0, 0x8, 0x8, 0x8, 0x56208e6d8de0, 0x1, 0xc4784aecd0)
	/usr/lib/golang/src/io/io.go:307 +0xab fp=0xc420d755d8 sp=0xc420d75578
io.ReadFull(0x56208ec32ae0, 0xc47a169100, 0xc4784aecd0, 0x8, 0x8, 0xc420069939, 0x1, 0xc4784aecc1)
	/usr/lib/golang/src/io/io.go:325 +0x5a fp=0xc420d75630 sp=0xc420d755d8
encoding/binary.Read(0x56208ec32ae0, 0xc47a169100, 0x56208ec418a0, 0x56208ec8be78, 0x56208e6b45c0, 0xc4784aecc8, 0x56208db15f01, 0xc420d75828)
	/usr/lib/golang/src/encoding/binary/binary.go:170 +0xec fp=0xc420d757b8 sp=0xc420d75630
github.com/coreos/etcd/rafthttp.(*messageDecoder).decode(0xc451f89cf0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/msg_codec.go:53 +0x11f fp=0xc420d75950 sp=0xc420d757b8
github.com/coreos/etcd/rafthttp.(*streamReader).decodeLoop(0xc42001c680, 0x56208ec37020, 0xc47a169100, 0x56208e310210, 0x7, 0x2, 0x0)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/stream.go:358 +0x293 fp=0xc420d75df8 sp=0xc420d75950
github.com/coreos/etcd/rafthttp.(*streamReader).run(0xc42001c680)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/stream.go:311 +0x69f fp=0xc420d75fd8 sp=0xc420d75df8
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420d75fe0 sp=0xc420d75fd8
created by github.com/coreos/etcd/rafthttp.(*streamReader).start
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/stream.go:296 +0xb9

goroutine 114 [select]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc479081bd0 sp=0xc479081ba0
runtime.selectgoImpl(0xc479081f20, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc479081e48 sp=0xc479081bd0
runtime.selectgo(0xc479081f20)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc479081e70 sp=0xc479081e48
github.com/xiang90/probing.(*prober).AddHTTP.func1(0xc420070600, 0xc420068370, 0x1, 0x1, 0xc42000c520, 0xc420095270)
	/usr/share/gocode/src/github.com/xiang90/probing/prober.go:55 +0x497 fp=0xc479081fb0 sp=0xc479081e70
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc479081fb8 sp=0xc479081fb0
created by github.com/xiang90/probing.(*prober).AddHTTP
	/usr/share/gocode/src/github.com/xiang90/probing/prober.go:85 +0x1db

goroutine 115 [select]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc4209f2ba8 sp=0xc4209f2b78
runtime.selectgoImpl(0xc4209f2f30, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc4209f2e20 sp=0xc4209f2ba8
runtime.selectgo(0xc4209f2f30)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc4209f2e48 sp=0xc4209f2e20
github.com/coreos/etcd/rafthttp.monitorProbingStatus(0x56208ec416c0, 0xc420095270, 0xc420068340, 0x10)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/probing_status.go:51 +0x508 fp=0xc4209f2fc0 sp=0xc4209f2e48
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc4209f2fc8 sp=0xc4209f2fc0
created by github.com/coreos/etcd/rafthttp.addPeerToProber
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/probing_status.go:43 +0x2d2

goroutine 116 [select]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc42067f8f0 sp=0xc42067f8c0
runtime.selectgoImpl(0xc42067fea8, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc42067fb68 sp=0xc42067f8f0
runtime.selectgo(0xc42067fea8)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc42067fb90 sp=0xc42067fb68
github.com/coreos/etcd/etcdserver.(*EtcdServer).run(0xc42006b8c0)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/etcdserver/server.go:712 +0x9d6 fp=0xc42067ffd8 sp=0xc42067fb90
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc42067ffe0 sp=0xc42067ffd8
created by github.com/coreos/etcd/etcdserver.(*EtcdServer).start
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/etcdserver/server.go:542 +0x4ab

goroutine 149 [select, 383 minutes, locked to thread]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc420338478 sp=0xc420338448
runtime.selectgoImpl(0xc420338750, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc4203386f0 sp=0xc420338478
runtime.selectgo(0xc420338750)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc420338718 sp=0xc4203386f0
runtime.ensureSigM.func1()
	/usr/lib/golang/src/runtime/signal_unix.go:434 +0x2e3 fp=0xc4203387e0 sp=0xc420338718
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc4203387e8 sp=0xc4203387e0
created by runtime.ensureSigM
	/usr/lib/golang/src/runtime/signal_unix.go:447 +0xdc

goroutine 118 [select, 383 minutes]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc420682b90 sp=0xc420682b60
runtime.selectgoImpl(0xc420682ec8, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc420682e08 sp=0xc420682b90
runtime.selectgo(0xc420682ec8)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc420682e30 sp=0xc420682e08
github.com/coreos/etcd/etcdserver.(*EtcdServer).purgeFile(0xc42006b8c0)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/etcdserver/server.go:553 +0x2aa fp=0xc420682f90 sp=0xc420682e30
github.com/coreos/etcd/etcdserver.(*EtcdServer).(github.com/coreos/etcd/etcdserver.purgeFile)-fm()
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/etcdserver/server.go:514 +0x2c fp=0xc420682fa8 sp=0xc420682f90
github.com/coreos/etcd/etcdserver.(*EtcdServer).goAttach.func1(0xc42006b8c0, 0xc420069bc0)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/etcdserver/server.go:1638 +0x57 fp=0xc420682fd0 sp=0xc420682fa8
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420682fd8 sp=0xc420682fd0
created by github.com/coreos/etcd/etcdserver.(*EtcdServer).goAttach
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/etcdserver/server.go:1639 +0x1aa

goroutine 119 [select]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc420d69bc8 sp=0xc420d69b98
runtime.selectgoImpl(0xc420d69f00, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc420d69e40 sp=0xc420d69bc8
runtime.selectgo(0xc420d69f00)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc420d69e68 sp=0xc420d69e40
github.com/coreos/etcd/etcdserver.monitorFileDescriptor(0xc420021260)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/etcdserver/metrics.go:89 +0x1af fp=0xc420d69f90 sp=0xc420d69e68
github.com/coreos/etcd/etcdserver.(*EtcdServer).Start.func2()
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/etcdserver/server.go:515 +0x33 fp=0xc420d69fa8 sp=0xc420d69f90
github.com/coreos/etcd/etcdserver.(*EtcdServer).goAttach.func1(0xc42006b8c0, 0xc420069bd0)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/etcdserver/server.go:1638 +0x57 fp=0xc420d69fd0 sp=0xc420d69fa8
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420d69fd8 sp=0xc420d69fd0
created by github.com/coreos/etcd/etcdserver.(*EtcdServer).goAttach
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/etcdserver/server.go:1639 +0x1aa

goroutine 120 [select]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc42067eba0 sp=0xc42067eb70
runtime.selectgoImpl(0xc42067eec8, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc42067ee18 sp=0xc42067eba0
runtime.selectgo(0xc42067eec8)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc42067ee40 sp=0xc42067ee18
github.com/coreos/etcd/etcdserver.(*EtcdServer).monitorVersions(0xc42006b8c0)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/etcdserver/server.go:1488 +0x4b6 fp=0xc42067ef90 sp=0xc42067ee40
github.com/coreos/etcd/etcdserver.(*EtcdServer).(github.com/coreos/etcd/etcdserver.monitorVersions)-fm()
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/etcdserver/server.go:516 +0x2c fp=0xc42067efa8 sp=0xc42067ef90
github.com/coreos/etcd/etcdserver.(*EtcdServer).goAttach.func1(0xc42006b8c0, 0xc420069cc0)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/etcdserver/server.go:1638 +0x57 fp=0xc42067efd0 sp=0xc42067efa8
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc42067efd8 sp=0xc42067efd0
created by github.com/coreos/etcd/etcdserver.(*EtcdServer).goAttach
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/etcdserver/server.go:1639 +0x1aa

goroutine 121 [select, 383 minutes]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc420d859e0 sp=0xc420d859b0
runtime.selectgoImpl(0xc420d85e48, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc420d85c58 sp=0xc420d859e0
runtime.selectgo(0xc420d85e48)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc420d85c80 sp=0xc420d85c58
github.com/coreos/etcd/etcdserver.(*EtcdServer).linearizableReadLoop(0xc42006b8c0)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/etcdserver/v3_server.go:722 +0xae6 fp=0xc420d85f90 sp=0xc420d85c80
github.com/coreos/etcd/etcdserver.(*EtcdServer).(github.com/coreos/etcd/etcdserver.linearizableReadLoop)-fm()
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/etcdserver/server.go:517 +0x2c fp=0xc420d85fa8 sp=0xc420d85f90
github.com/coreos/etcd/etcdserver.(*EtcdServer).goAttach.func1(0xc42006b8c0, 0xc420069cd0)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/etcdserver/server.go:1638 +0x57 fp=0xc420d85fd0 sp=0xc420d85fa8
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420d85fd8 sp=0xc420d85fd0
created by github.com/coreos/etcd/etcdserver.(*EtcdServer).goAttach
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/etcdserver/server.go:1639 +0x1aa

goroutine 122 [IO wait, 3 minutes]:
runtime.gopark(0x56208e862580, 0x7fdc68ae2e68, 0x56208e30f82a, 0x7, 0xc42067dc1b, 0x5)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc42067dc20 sp=0xc42067dbf0
runtime.netpollblock(0x7fdc68ae2e40, 0x72, 0x56208ec35020)
	/usr/lib/golang/src/runtime/netpoll.go:345 +0xc9 fp=0xc42067dc68 sp=0xc42067dc20
net.runtime_pollWait(0x7fdc68ae2e40, 0x72, 0x0)
	/usr/lib/golang/src/runtime/netpoll.go:164 +0x5b fp=0xc42067dc90 sp=0xc42067dc68
net.(*pollDesc).wait(0xc4201ec458, 0x72, 0x0, 0xc43681a160)
	/usr/lib/golang/src/net/fd_poll_runtime.go:75 +0x3a fp=0xc42067dcb8 sp=0xc42067dc90
net.(*pollDesc).waitRead(0xc4201ec458, 0xffffffffffffffff, 0x0)
	/usr/lib/golang/src/net/fd_poll_runtime.go:80 +0x36 fp=0xc42067dce8 sp=0xc42067dcb8
net.(*netFD).accept(0xc4201ec3f0, 0x0, 0x56208ec330e0, 0xc43681a160)
	/usr/lib/golang/src/net/fd_unix.go:430 +0x1e7 fp=0xc42067dd90 sp=0xc42067dce8
net.(*TCPListener).accept(0xc42007c200, 0xc432dfd480, 0x56208e703800, 0xc42067de28)
	/usr/lib/golang/src/net/tcpsock_posix.go:136 +0x30 fp=0xc42067ddc0 sp=0xc42067dd90
net.(*TCPListener).Accept(0xc42007c200, 0x56208dae025b, 0xc42067de30, 0x56208dd2b010, 0x56208db03050)
	/usr/lib/golang/src/net/tcpsock.go:228 +0x4b fp=0xc42067de10 sp=0xc42067ddc0
github.com/coreos/etcd/pkg/transport.(*rwTimeoutListener).Accept(0xc4201f4780, 0x56208e862008, 0xc432dfd400, 0x56208ec3d960, 0xc42001bc80)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/pkg/transport/timeout_listener.go:49 +0x39 fp=0xc42067de68 sp=0xc42067de10
net/http.(*Server).Serve(0xc42012ae70, 0x56208ec3b2e0, 0xc4201f4780, 0x0, 0x0)
	/usr/lib/golang/src/net/http/server.go:2643 +0x22a fp=0xc42067df10 sp=0xc42067de68
github.com/coreos/etcd/embed.servePeerHTTP(0x56208ec3b2e0, 0xc4201f4780, 0x56208ec32a20, 0xc4201823c0, 0x0, 0x0)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/embed/serve.go:142 +0x168 fp=0xc42067df68 sp=0xc42067df10
github.com/coreos/etcd/embed.(*Etcd).serve.func1(0xc42006b600, 0x56208ec32a20, 0xc4201823c0, 0x56208ec3b2e0, 0xc4201f4780)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/embed/etcd.go:317 +0x4f fp=0xc42067dfb8 sp=0xc42067df68
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc42067dfc0 sp=0xc42067dfb8
created by github.com/coreos/etcd/embed.(*Etcd).serve
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/embed/etcd.go:318 +0x17b

goroutine 170 [select]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc420d7fb70 sp=0xc420d7fb40
runtime.selectgoImpl(0xc420d7ff40, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc420d7fde8 sp=0xc420d7fb70
runtime.selectgo(0xc420d7ff40)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc420d7fe10 sp=0xc420d7fde8
github.com/coreos/etcd/etcdserver/api/v3rpc.monitorLeader.func1(0xc42006b8c0, 0xc4200695c0)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/etcdserver/api/v3rpc/interceptor.go:115 +0x43e fp=0xc420d7ffd0 sp=0xc420d7fe10
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420d7ffd8 sp=0xc420d7ffd0
created by github.com/coreos/etcd/etcdserver/api/v3rpc.monitorLeader
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/etcdserver/api/v3rpc/interceptor.go:141 +0xc1

goroutine 127 [IO wait]:
runtime.gopark(0x56208e862580, 0x7fdc68ae2da8, 0x56208e30f82a, 0x7, 0xc420043a1b, 0x5)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc420043980 sp=0xc420043950
runtime.netpollblock(0x7fdc68ae2d80, 0x72, 0x56208ec35020)
	/usr/lib/golang/src/runtime/netpoll.go:345 +0xc9 fp=0xc4200439c8 sp=0xc420043980
net.runtime_pollWait(0x7fdc68ae2d80, 0x72, 0x0)
	/usr/lib/golang/src/runtime/netpoll.go:164 +0x5b fp=0xc4200439f0 sp=0xc4200439c8
net.(*pollDesc).wait(0xc4201ec538, 0x72, 0x0, 0xc420a0d040)
	/usr/lib/golang/src/net/fd_poll_runtime.go:75 +0x3a fp=0xc420043a18 sp=0xc4200439f0
net.(*pollDesc).waitRead(0xc4201ec538, 0xffffffffffffffff, 0x0)
	/usr/lib/golang/src/net/fd_poll_runtime.go:80 +0x36 fp=0xc420043a48 sp=0xc420043a18
net.(*netFD).accept(0xc4201ec4d0, 0x0, 0x56208ec330e0, 0xc420a0d040)
	/usr/lib/golang/src/net/fd_unix.go:430 +0x1e7 fp=0xc420043af0 sp=0xc420043a48
net.(*TCPListener).accept(0xc42007c208, 0x56208dab008d, 0x56208e6c2560, 0xc4201fa000)
	/usr/lib/golang/src/net/tcpsock_posix.go:136 +0x30 fp=0xc420043b20 sp=0xc420043af0
net.(*TCPListener).Accept(0xc42007c208, 0xc4201fa000, 0xc420043b98, 0x0, 0x56208e73f060)
	/usr/lib/golang/src/net/tcpsock.go:228 +0x4b fp=0xc420043b70 sp=0xc420043b20
github.com/coreos/etcd/pkg/transport.(*limitListener).Accept(0xc4201f4820, 0x56208dae025b, 0xc420043bf8, 0x56208e29b7be, 0x56208db03050)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/pkg/transport/limit_listen.go:46 +0x61 fp=0xc420043bd8 sp=0xc420043b70
github.com/coreos/etcd/pkg/transport.(*keepaliveListener).Accept(0xc4201e6f00, 0x56208e85f410, 0xc4201f25c0, 0x56208ec41c60, 0xc48880ce70)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/pkg/transport/keepalive_listener.go:49 +0x39 fp=0xc420043c30 sp=0xc420043bd8
github.com/cockroachdb/cmux.(*cMux).Serve(0xc4201f25c0, 0x0, 0x0)
	/usr/share/gocode/src/github.com/cockroachdb/cmux/cmux.go:110 +0x97 fp=0xc420043ca0 sp=0xc420043c30
github.com/coreos/etcd/embed.(*serveCtx).serve(0xc4201f2340, 0xc42006b8c0, 0x0, 0x56208ec31820, 0xc4200d3820, 0xc420021140, 0xc42033df90, 0xc42001e8c0)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/embed/serve.go:119 +0x22a fp=0xc420043f58 sp=0xc420043ca0
github.com/coreos/etcd/embed.(*Etcd).serve.func2(0xc42006b600, 0xc42007c058, 0x56208ec31820, 0xc4200d3820, 0xc4201f2340)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/embed/etcd.go:330 +0x6f fp=0xc420043fb8 sp=0xc420043f58
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420043fc0 sp=0xc420043fb8
created by github.com/coreos/etcd/embed.(*Etcd).serve
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/embed/etcd.go:331 +0x340

goroutine 128 [select]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc4209ef500 sp=0xc4209ef4d0
runtime.selectgoImpl(0xc4209efcc0, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc4209ef778 sp=0xc4209ef500
runtime.selectgo(0xc4209efcc0)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc4209ef7a0 sp=0xc4209ef778
github.com/coreos/etcd/etcdserver.(*raftNode).start.func1(0xc42006b8f0, 0xc4201a6160, 0x3b9aca00)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/etcdserver/raft.go:145 +0xd72 fp=0xc4209effc8 sp=0xc4209ef7a0
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc4209effd0 sp=0xc4209effc8
created by github.com/coreos/etcd/etcdserver.(*raftNode).start
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/etcdserver/raft.go:233 +0x108

goroutine 129 [select]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc4210f3c50 sp=0xc4210f3c20
runtime.selectgoImpl(0xc4210f3f48, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc4210f3ec8 sp=0xc4210f3c50
runtime.selectgo(0xc4210f3f48)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc4210f3ef0 sp=0xc4210f3ec8
github.com/coreos/etcd/pkg/schedule.(*fifo).run(0xc4200216e0)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/pkg/schedule/schedule.go:146 +0x35e fp=0xc4210f3fd8 sp=0xc4210f3ef0
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc4210f3fe0 sp=0xc4210f3fd8
created by github.com/coreos/etcd/pkg/schedule.NewFIFOScheduler
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/pkg/schedule/schedule.go:71 +0x1ae

goroutine 146 [select]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc420d84af8 sp=0xc420d84ac8
runtime.selectgoImpl(0xc420d84f08, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc420d84d70 sp=0xc420d84af8
runtime.selectgo(0xc420d84f08)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc420d84d98 sp=0xc420d84d70
github.com/coreos/etcd/pkg/fileutil.purgeFile.func1(0xc420183830, 0x22, 0xc4200218c0, 0x56208e30d5a3, 0x4, 0x5, 0x0, 0x6fc23ac00, 0xc4200211a0)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/pkg/fileutil/purge.go:70 +0x83c fp=0xc420d84f98 sp=0xc420d84d98
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420d84fa0 sp=0xc420d84f98
created by github.com/coreos/etcd/pkg/fileutil.purgeFile
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/pkg/fileutil/purge.go:76 +0xc5

goroutine 147 [select]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc420683af8 sp=0xc420683ac8
runtime.selectgoImpl(0xc420683f08, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc420683d70 sp=0xc420683af8
runtime.selectgo(0xc420683f08)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc420683d98 sp=0xc420683d70
github.com/coreos/etcd/pkg/fileutil.purgeFile.func1(0xc420183890, 0x21, 0xc420021920, 0x56208e30cdc5, 0x3, 0x5, 0x0, 0x6fc23ac00, 0xc4200211a0)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/pkg/fileutil/purge.go:70 +0x83c fp=0xc420683f98 sp=0xc420683d98
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420683fa0 sp=0xc420683f98
created by github.com/coreos/etcd/pkg/fileutil.purgeFile
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/pkg/fileutil/purge.go:76 +0xc5

goroutine 504106 [IO wait]:
runtime.gopark(0x56208e862580, 0x7fd9de8f7e28, 0x56208e30f82a, 0x7, 0x56208db23d1b, 0x5)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc451e41838 sp=0xc451e41808
runtime.netpollblock(0x7fd9de8f7e00, 0x72, 0xb)
	/usr/lib/golang/src/runtime/netpoll.go:345 +0xc9 fp=0xc451e41880 sp=0xc451e41838
net.runtime_pollWait(0x7fd9de8f7e00, 0x72, 0x26)
	/usr/lib/golang/src/runtime/netpoll.go:164 +0x5b fp=0xc451e418a8 sp=0xc451e41880
net.(*pollDesc).wait(0xc432b313a8, 0x72, 0x56208ec35020, 0x56208ec2ebc0)
	/usr/lib/golang/src/net/fd_poll_runtime.go:75 +0x3a fp=0xc451e418d0 sp=0xc451e418a8
net.(*pollDesc).waitRead(0xc432b313a8, 0xc4d9b61000, 0x1000)
	/usr/lib/golang/src/net/fd_poll_runtime.go:80 +0x36 fp=0xc451e41900 sp=0xc451e418d0
net.(*netFD).Read(0xc432b31340, 0xc4d9b61000, 0x1000, 0x1000, 0x0, 0x56208ec35020, 0x56208ec2ebc0)
	/usr/lib/golang/src/net/fd_unix.go:250 +0x1b9 fp=0xc451e41968 sp=0xc451e41900
net.(*conn).Read(0xc43905fc58, 0xc4d9b61000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/lib/golang/src/net/net.go:181 +0x72 fp=0xc451e419d8 sp=0xc451e41968
github.com/coreos/etcd/pkg/transport.timeoutConn.Read(0x56208ec41ea0, 0xc43905fc58, 0x0, 0x0, 0xc4d9b61000, 0x1000, 0x1000, 0x4, 0xc451e41a90, 0x56208dab0ac6)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/pkg/transport/timeout_conn.go:43 +0x5f fp=0xc451e41a20 sp=0xc451e419d8
github.com/coreos/etcd/pkg/transport.(*timeoutConn).Read(0xc4d9b53e20, 0xc4d9b61000, 0x1000, 0x1000, 0xc47d082360, 0xc47c35d360, 0xc451e41be5)
	<autogenerated>:42 +0x88 fp=0xc451e41a80 sp=0xc451e41a20
net/http.(*persistConn).Read(0xc432b39680, 0xc4d9b61000, 0x1000, 0x1000, 0x4, 0x56208db00940, 0xc47d082360)
	/usr/lib/golang/src/net/http/transport.go:1316 +0x14d fp=0xc451e41af8 sp=0xc451e41a80
bufio.(*Reader).fill(0xc4d9b5fd40)
	/usr/lib/golang/src/bufio/bufio.go:97 +0x119 fp=0xc451e41b58 sp=0xc451e41af8
bufio.(*Reader).Peek(0xc4d9b5fd40, 0x1, 0xc451e41be5, 0x1, 0x0, 0xc477d98f00, 0x0)
	/usr/lib/golang/src/bufio/bufio.go:129 +0x69 fp=0xc451e41b90 sp=0xc451e41b58
net/http.(*persistConn).readLoop(0xc432b39680)
	/usr/lib/golang/src/net/http/transport.go:1474 +0x198 fp=0xc451e41fd8 sp=0xc451e41b90
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc451e41fe0 sp=0xc451e41fd8
created by net/http.(*Transport).dialConn
	/usr/lib/golang/src/net/http/transport.go:1117 +0xa37

goroutine 140 [semacquire]:
runtime.gopark(0x56208e8625b8, 0xc4201f3f98, 0x56208e312e73, 0xa, 0xc4200f1d1a, 0x3)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc42003cc30 sp=0xc42003cc00
runtime.goparkunlock(0xc4201f3f98, 0x56208e312e73, 0xa, 0x1a, 0x3)
	/usr/lib/golang/src/runtime/proc.go:277 +0x60 fp=0xc42003cc70 sp=0xc42003cc30
sync.runtime_notifyListWait(0xc4201f3f90, 0x56200000165c)
	/usr/lib/golang/src/runtime/sema.go:297 +0x10f fp=0xc42003ccc0 sp=0xc42003cc70
sync.(*Cond).Wait(0xc4201f3f80)
	/usr/lib/golang/src/sync/cond.go:57 +0x8b fp=0xc42003ccf0 sp=0xc42003ccc0
net/http.(*connReader).abortPendingRead(0xc4201f3f00)
	/usr/lib/golang/src/net/http/server.go:686 +0xc1 fp=0xc42003cd30 sp=0xc42003ccf0
net/http.(*response).finishRequest(0xc486a38380)
	/usr/lib/golang/src/net/http/server.go:1543 +0xa6 fp=0xc42003cd58 sp=0xc42003cd30
net/http.(*conn).serve(0xc4201c0fa0, 0x56208ec3d8a0, 0xc4201f3ec0)
	/usr/lib/golang/src/net/http/server.go:1830 +0x654 fp=0xc42003cfc8 sp=0xc42003cd58
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc42003cfd0 sp=0xc42003cfc8
created by net/http.(*Server).Serve
	/usr/lib/golang/src/net/http/server.go:2668 +0x2d0

goroutine 150 [chan receive, 383 minutes]:
runtime.gopark(0x56208e8625b8, 0xc420021cd8, 0x56208e3141f2, 0xc, 0x56208db00a17, 0x3)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc42002be18 sp=0xc42002bde8
runtime.goparkunlock(0xc420021cd8, 0x56208e3141f2, 0xc, 0x56208ec8bd17, 0x3)
	/usr/lib/golang/src/runtime/proc.go:277 +0x60 fp=0xc42002be58 sp=0xc42002be18
runtime.chanrecv(0x56208e6c24e0, 0xc420021c80, 0xc42002bfb8, 0x1, 0xc42002bf50)
	/usr/lib/golang/src/runtime/chan.go:513 +0x375 fp=0xc42002bef8 sp=0xc42002be58
runtime.chanrecv1(0x56208e6c24e0, 0xc420021c80, 0xc42002bfb8)
	/usr/lib/golang/src/runtime/chan.go:395 +0x35 fp=0xc42002bf30 sp=0xc42002bef8
github.com/coreos/etcd/pkg/osutil.HandleInterrupts.func1(0xc420021c80)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/pkg/osutil/interrupt_unix.go:51 +0x73 fp=0xc42002bfd8 sp=0xc42002bf30
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc42002bfe0 sp=0xc42002bfd8
created by github.com/coreos/etcd/pkg/osutil.HandleInterrupts
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/pkg/osutil/interrupt_unix.go:72 +0xe8

goroutine 171 [chan receive, 383 minutes]:
runtime.gopark(0x56208e8625b8, 0xc420e8f5b8, 0x56208e3141f2, 0xc, 0xc420026017, 0x3)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc420d7d4f0 sp=0xc420d7d4c0
runtime.goparkunlock(0xc420e8f5b8, 0x56208e3141f2, 0xc, 0x8bf047d7cf7a3817, 0x3)
	/usr/lib/golang/src/runtime/proc.go:277 +0x60 fp=0xc420d7d530 sp=0xc420d7d4f0
runtime.chanrecv(0x56208e6c20a0, 0xc420e8f560, 0xc420d7d638, 0x14926f3b17be3001, 0x101)
	/usr/lib/golang/src/runtime/chan.go:513 +0x375 fp=0xc420d7d5d0 sp=0xc420d7d530
runtime.chanrecv2(0x56208e6c20a0, 0xc420e8f560, 0xc420d7d638, 0x56208db03050)
	/usr/lib/golang/src/runtime/chan.go:400 +0x35 fp=0xc420d7d608 sp=0xc420d7d5d0
github.com/cockroachdb/cmux.(*muxListener).Accept(0xc420159740, 0x56208e861a48, 0xc42012c000, 0x56208ec41a20, 0xc4201c1860)
	<autogenerated>:9 +0x7c fp=0xc420d7d658 sp=0xc420d7d608
google.golang.org/grpc.(*Server).Serve(0xc42012c000, 0x56208ec3c7e0, 0xc420159740, 0x0, 0x0)
	/usr/share/gocode/src/google.golang.org/grpc/server.go:354 +0x182 fp=0xc420d7d778 sp=0xc420d7d658
github.com/coreos/etcd/embed.(*serveCtx).serve.func1(0xc420021140, 0xc42012c000, 0x56208ec3c7e0, 0xc420159740)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/embed/serve.go:70 +0x41 fp=0xc420d7d7c0 sp=0xc420d7d778
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420d7d7c8 sp=0xc420d7d7c0
created by github.com/coreos/etcd/embed.(*serveCtx).serve
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/embed/serve.go:70 +0x9ea

goroutine 174 [chan receive, 383 minutes]:
runtime.gopark(0x56208e8625b8, 0xc420067fd8, 0x56208e3141f2, 0xc, 0xc4210e6017, 0x3)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc420336e40 sp=0xc420336e10
runtime.goparkunlock(0xc420067fd8, 0x56208e3141f2, 0xc, 0xc4201d0817, 0x3)
	/usr/lib/golang/src/runtime/proc.go:277 +0x60 fp=0xc420336e80 sp=0xc420336e40
runtime.chanrecv(0x56208e6c13e0, 0xc420067f80, 0x0, 0xc420e8e901, 0x0)
	/usr/lib/golang/src/runtime/chan.go:513 +0x375 fp=0xc420336f20 sp=0xc420336e80
runtime.chanrecv1(0x56208e6c13e0, 0xc420067f80, 0x0)
	/usr/lib/golang/src/runtime/chan.go:395 +0x35 fp=0xc420336f58 sp=0xc420336f20
github.com/coreos/etcd/etcdserver/etcdserverpb.RegisterKVHandlerFromEndpoint.func1.1(0x7fdc68b2a3e8, 0xc4201f2300, 0xc42012eea0, 0xc4200698d0, 0x9)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.gw.go:586 +0x5a fp=0xc420336fb8 sp=0xc420336f58
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420336fc0 sp=0xc420336fb8
created by github.com/coreos/etcd/etcdserver/etcdserverpb.RegisterKVHandlerFromEndpoint.func1
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.gw.go:590 +0x1b9

goroutine 173 [select, 383 minutes]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc4209edab0 sp=0xc4209eda80
runtime.selectgoImpl(0xc4209edeb8, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc4209edd28 sp=0xc4209edab0
runtime.selectgo(0xc4209edeb8)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc4209edd50 sp=0xc4209edd28
google.golang.org/grpc.(*addrConn).transportMonitor(0xc4200969a0)
	/usr/share/gocode/src/google.golang.org/grpc/clientconn.go:751 +0x668 fp=0xc4209edf80 sp=0xc4209edd50
google.golang.org/grpc.(*ClientConn).resetAddrConn.func1(0xc4200969a0)
	/usr/share/gocode/src/google.golang.org/grpc/clientconn.go:506 +0x1db fp=0xc4209edfd8 sp=0xc4209edf80
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc4209edfe0 sp=0xc4209edfd8
created by google.golang.org/grpc.(*ClientConn).resetAddrConn
	/usr/share/gocode/src/google.golang.org/grpc/clientconn.go:507 +0x374

goroutine 177 [chan receive, 383 minutes]:
runtime.gopark(0x56208e8625b8, 0xc420067fd8, 0x56208e3141f2, 0xc, 0xc4210e6017, 0x3)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc42033a640 sp=0xc42033a610
runtime.goparkunlock(0xc420067fd8, 0x56208e3141f2, 0xc, 0xc4201d0717, 0x3)
	/usr/lib/golang/src/runtime/proc.go:277 +0x60 fp=0xc42033a680 sp=0xc42033a640
runtime.chanrecv(0x56208e6c13e0, 0xc420067f80, 0x0, 0xc420e8e701, 0x0)
	/usr/lib/golang/src/runtime/chan.go:513 +0x375 fp=0xc42033a720 sp=0xc42033a680
runtime.chanrecv1(0x56208e6c13e0, 0xc420067f80, 0x0)
	/usr/lib/golang/src/runtime/chan.go:395 +0x35 fp=0xc42033a758 sp=0xc42033a720
github.com/coreos/etcd/etcdserver/etcdserverpb.RegisterWatchHandlerFromEndpoint.func1.1(0x7fdc68b2a3e8, 0xc4201f2300, 0xc42012fc20, 0xc4200698d0, 0x9)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.gw.go:783 +0x5a fp=0xc42033a7b8 sp=0xc42033a758
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc42033a7c0 sp=0xc42033a7b8
created by github.com/coreos/etcd/etcdserver/etcdserverpb.RegisterWatchHandlerFromEndpoint.func1
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.gw.go:787 +0x1b9

goroutine 176 [select, 383 minutes]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc4209ebab0 sp=0xc4209eba80
runtime.selectgoImpl(0xc4209ebeb8, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc4209ebd28 sp=0xc4209ebab0
runtime.selectgo(0xc4209ebeb8)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc4209ebd50 sp=0xc4209ebd28
google.golang.org/grpc.(*addrConn).transportMonitor(0xc420097760)
	/usr/share/gocode/src/google.golang.org/grpc/clientconn.go:751 +0x668 fp=0xc4209ebf80 sp=0xc4209ebd50
google.golang.org/grpc.(*ClientConn).resetAddrConn.func1(0xc420097760)
	/usr/share/gocode/src/google.golang.org/grpc/clientconn.go:506 +0x1db fp=0xc4209ebfd8 sp=0xc4209ebf80
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc4209ebfe0 sp=0xc4209ebfd8
created by google.golang.org/grpc.(*ClientConn).resetAddrConn
	/usr/share/gocode/src/google.golang.org/grpc/clientconn.go:507 +0x374

goroutine 180 [chan receive, 383 minutes]:
runtime.gopark(0x56208e8625b8, 0xc420067fd8, 0x56208e3141f2, 0xc, 0xc4210e6217, 0x3)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc420337640 sp=0xc420337610
runtime.goparkunlock(0xc420067fd8, 0x56208e3141f2, 0xc, 0x17, 0x3)
	/usr/lib/golang/src/runtime/proc.go:277 +0x60 fp=0xc420337680 sp=0xc420337640
runtime.chanrecv(0x56208e6c13e0, 0xc420067f80, 0x0, 0x1, 0x0)
	/usr/lib/golang/src/runtime/chan.go:513 +0x375 fp=0xc420337720 sp=0xc420337680
runtime.chanrecv1(0x56208e6c13e0, 0xc420067f80, 0x0)
	/usr/lib/golang/src/runtime/chan.go:395 +0x35 fp=0xc420337758 sp=0xc420337720
github.com/coreos/etcd/etcdserver/etcdserverpb.RegisterLeaseHandlerFromEndpoint.func1.1(0x7fdc68b2a3e8, 0xc4201f2300, 0xc42012fd40, 0xc4200698d0, 0x9)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.gw.go:852 +0x5a fp=0xc4203377b8 sp=0xc420337758
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc4203377c0 sp=0xc4203377b8
created by github.com/coreos/etcd/etcdserver/etcdserverpb.RegisterLeaseHandlerFromEndpoint.func1
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.gw.go:856 +0x1b9

goroutine 179 [select, 383 minutes]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc420f97ab0 sp=0xc420f97a80
runtime.selectgoImpl(0xc420f97eb8, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc420f97d28 sp=0xc420f97ab0
runtime.selectgo(0xc420f97eb8)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc420f97d50 sp=0xc420f97d28
google.golang.org/grpc.(*addrConn).transportMonitor(0xc420097a20)
	/usr/share/gocode/src/google.golang.org/grpc/clientconn.go:751 +0x668 fp=0xc420f97f80 sp=0xc420f97d50
google.golang.org/grpc.(*ClientConn).resetAddrConn.func1(0xc420097a20)
	/usr/share/gocode/src/google.golang.org/grpc/clientconn.go:506 +0x1db fp=0xc420f97fd8 sp=0xc420f97f80
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420f97fe0 sp=0xc420f97fd8
created by google.golang.org/grpc.(*ClientConn).resetAddrConn
	/usr/share/gocode/src/google.golang.org/grpc/clientconn.go:507 +0x374

goroutine 183 [chan receive, 383 minutes]:
runtime.gopark(0x56208e8625b8, 0xc420067fd8, 0x56208e3141f2, 0xc, 0x17, 0x3)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc420d79640 sp=0xc420d79610
runtime.goparkunlock(0xc420067fd8, 0x56208e3141f2, 0xc, 0x17, 0x3)
	/usr/lib/golang/src/runtime/proc.go:277 +0x60 fp=0xc420d79680 sp=0xc420d79640
runtime.chanrecv(0x56208e6c13e0, 0xc420067f80, 0x0, 0x1, 0x0)
	/usr/lib/golang/src/runtime/chan.go:513 +0x375 fp=0xc420d79720 sp=0xc420d79680
runtime.chanrecv1(0x56208e6c13e0, 0xc420067f80, 0x0)
	/usr/lib/golang/src/runtime/chan.go:395 +0x35 fp=0xc420d79758 sp=0xc420d79720
github.com/coreos/etcd/etcdserver/etcdserverpb.RegisterClusterHandlerFromEndpoint.func1.1(0x7fdc68b2a3e8, 0xc4201f2300, 0xc42012fe60, 0xc4200698d0, 0x9)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.gw.go:1017 +0x5a fp=0xc420d797b8 sp=0xc420d79758
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420d797c0 sp=0xc420d797b8
created by github.com/coreos/etcd/etcdserver/etcdserverpb.RegisterClusterHandlerFromEndpoint.func1
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.gw.go:1021 +0x1b9

goroutine 182 [select, 383 minutes]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc420f9bab0 sp=0xc420f9ba80
runtime.selectgoImpl(0xc420f9beb8, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc420f9bd28 sp=0xc420f9bab0
runtime.selectgo(0xc420f9beb8)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc420f9bd50 sp=0xc420f9bd28
google.golang.org/grpc.(*addrConn).transportMonitor(0xc420097ce0)
	/usr/share/gocode/src/google.golang.org/grpc/clientconn.go:751 +0x668 fp=0xc420f9bf80 sp=0xc420f9bd50
google.golang.org/grpc.(*ClientConn).resetAddrConn.func1(0xc420097ce0)
	/usr/share/gocode/src/google.golang.org/grpc/clientconn.go:506 +0x1db fp=0xc420f9bfd8 sp=0xc420f9bf80
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420f9bfe0 sp=0xc420f9bfd8
created by google.golang.org/grpc.(*ClientConn).resetAddrConn
	/usr/share/gocode/src/google.golang.org/grpc/clientconn.go:507 +0x374

goroutine 186 [chan receive, 383 minutes]:
runtime.gopark(0x56208e8625b8, 0xc420067fd8, 0x56208e3141f2, 0xc, 0xc42005c517, 0x3)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc420d7a640 sp=0xc420d7a610
runtime.goparkunlock(0xc420067fd8, 0x56208e3141f2, 0xc, 0x17, 0x3)
	/usr/lib/golang/src/runtime/proc.go:277 +0x60 fp=0xc420d7a680 sp=0xc420d7a640
runtime.chanrecv(0x56208e6c13e0, 0xc420067f80, 0x0, 0x1, 0x0)
	/usr/lib/golang/src/runtime/chan.go:513 +0x375 fp=0xc420d7a720 sp=0xc420d7a680
runtime.chanrecv1(0x56208e6c13e0, 0xc420067f80, 0x0)
	/usr/lib/golang/src/runtime/chan.go:395 +0x35 fp=0xc420d7a758 sp=0xc420d7a720
github.com/coreos/etcd/etcdserver/etcdserverpb.RegisterMaintenanceHandlerFromEndpoint.func1.1(0x7fdc68b2a3e8, 0xc4201f2300, 0xc4210be000, 0xc4200698d0, 0x9)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.gw.go:1182 +0x5a fp=0xc420d7a7b8 sp=0xc420d7a758
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420d7a7c0 sp=0xc420d7a7b8
created by github.com/coreos/etcd/etcdserver/etcdserverpb.RegisterMaintenanceHandlerFromEndpoint.func1
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.gw.go:1186 +0x1b9

goroutine 185 [select, 383 minutes]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc420d73ab0 sp=0xc420d73a80
runtime.selectgoImpl(0xc420d73eb8, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc420d73d28 sp=0xc420d73ab0
runtime.selectgo(0xc420d73eb8)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc420d73d50 sp=0xc420d73d28
google.golang.org/grpc.(*addrConn).transportMonitor(0xc420fa6580)
	/usr/share/gocode/src/google.golang.org/grpc/clientconn.go:751 +0x668 fp=0xc420d73f80 sp=0xc420d73d50
google.golang.org/grpc.(*ClientConn).resetAddrConn.func1(0xc420fa6580)
	/usr/share/gocode/src/google.golang.org/grpc/clientconn.go:506 +0x1db fp=0xc420d73fd8 sp=0xc420d73f80
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420d73fe0 sp=0xc420d73fd8
created by google.golang.org/grpc.(*ClientConn).resetAddrConn
	/usr/share/gocode/src/google.golang.org/grpc/clientconn.go:507 +0x374

goroutine 189 [chan receive, 383 minutes]:
runtime.gopark(0x56208e8625b8, 0xc420067fd8, 0x56208e3141f2, 0xc, 0xc42005c717, 0x3)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc420d7b640 sp=0xc420d7b610
runtime.goparkunlock(0xc420067fd8, 0x56208e3141f2, 0xc, 0x17, 0x3)
	/usr/lib/golang/src/runtime/proc.go:277 +0x60 fp=0xc420d7b680 sp=0xc420d7b640
runtime.chanrecv(0x56208e6c13e0, 0xc420067f80, 0x0, 0x1, 0x0)
	/usr/lib/golang/src/runtime/chan.go:513 +0x375 fp=0xc420d7b720 sp=0xc420d7b680
runtime.chanrecv1(0x56208e6c13e0, 0xc420067f80, 0x0)
	/usr/lib/golang/src/runtime/chan.go:395 +0x35 fp=0xc420d7b758 sp=0xc420d7b720
github.com/coreos/etcd/etcdserver/etcdserverpb.RegisterAuthHandlerFromEndpoint.func1.1(0x7fdc68b2a3e8, 0xc4201f2300, 0xc4210be120, 0xc4200698d0, 0x9)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.gw.go:1379 +0x5a fp=0xc420d7b7b8 sp=0xc420d7b758
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420d7b7c0 sp=0xc420d7b7b8
created by github.com/coreos/etcd/etcdserver/etcdserverpb.RegisterAuthHandlerFromEndpoint.func1
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.gw.go:1383 +0x1b9

goroutine 188 [select, 383 minutes]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc4210f1ab0 sp=0xc4210f1a80
runtime.selectgoImpl(0xc4210f1eb8, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc4210f1d28 sp=0xc4210f1ab0
runtime.selectgo(0xc4210f1eb8)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc4210f1d50 sp=0xc4210f1d28
google.golang.org/grpc.(*addrConn).transportMonitor(0xc420fa6840)
	/usr/share/gocode/src/google.golang.org/grpc/clientconn.go:751 +0x668 fp=0xc4210f1f80 sp=0xc4210f1d50
google.golang.org/grpc.(*ClientConn).resetAddrConn.func1(0xc420fa6840)
	/usr/share/gocode/src/google.golang.org/grpc/clientconn.go:506 +0x1db fp=0xc4210f1fd8 sp=0xc4210f1f80
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc4210f1fe0 sp=0xc4210f1fd8
created by google.golang.org/grpc.(*ClientConn).resetAddrConn
	/usr/share/gocode/src/google.golang.org/grpc/clientconn.go:507 +0x374

goroutine 152 [IO wait, 383 minutes]:
runtime.gopark(0x56208e862580, 0x7fdc68ae2328, 0x56208e30f82a, 0x7, 0x56208db23d1b, 0x5)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc420680ad0 sp=0xc420680aa0
runtime.netpollblock(0x7fdc68ae2300, 0x72, 0xb)
	/usr/lib/golang/src/runtime/netpoll.go:345 +0xc9 fp=0xc420680b18 sp=0xc420680ad0
net.runtime_pollWait(0x7fdc68ae2300, 0x72, 0x18)
	/usr/lib/golang/src/runtime/netpoll.go:164 +0x5b fp=0xc420680b40 sp=0xc420680b18
net.(*pollDesc).wait(0xc42001eae8, 0x72, 0x56208ec35020, 0x56208ec2ebc0)
	/usr/lib/golang/src/net/fd_poll_runtime.go:75 +0x3a fp=0xc420680b68 sp=0xc420680b40
net.(*pollDesc).waitRead(0xc42001eae8, 0xc4210c2000, 0x8000)
	/usr/lib/golang/src/net/fd_poll_runtime.go:80 +0x36 fp=0xc420680b98 sp=0xc420680b68
net.(*netFD).Read(0xc42001ea80, 0xc4210c2000, 0x8000, 0x8000, 0x0, 0x56208ec35020, 0x56208ec2ebc0)
	/usr/lib/golang/src/net/fd_unix.go:250 +0x1b9 fp=0xc420680c00 sp=0xc420680b98
net.(*conn).Read(0xc42007c178, 0xc4210c2000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/lib/golang/src/net/net.go:181 +0x72 fp=0xc420680c70 sp=0xc420680c00
bufio.(*Reader).Read(0xc4210c0120, 0xc4201849f8, 0x9, 0x9, 0x9, 0x0, 0x18)
	/usr/lib/golang/src/bufio/bufio.go:213 +0x314 fp=0xc420680d00 sp=0xc420680c70
io.ReadAtLeast(0x56208ec305e0, 0xc4210c0120, 0xc4201849f8, 0x9, 0x9, 0x9, 0xc420680dc0, 0x56208dabc56a, 0x30)
	/usr/lib/golang/src/io/io.go:307 +0xab fp=0xc420680d60 sp=0xc420680d00
io.ReadFull(0x56208ec305e0, 0xc4210c0120, 0xc4201849f8, 0x9, 0x9, 0xa0f9cc0000000000, 0xa000000000000000, 0x2)
	/usr/lib/golang/src/io/io.go:325 +0x5a fp=0xc420680db8 sp=0xc420680d60
golang.org/x/net/http2.readFrameHeader(0xc4201849f8, 0x9, 0x9, 0x56208ec305e0, 0xc4210c0120, 0x0, 0xc400000000, 0x0, 0x0)
	/usr/share/gocode/src/golang.org/x/net/http2/frame.go:237 +0x7d fp=0xc420680e18 sp=0xc420680db8
golang.org/x/net/http2.(*Framer).ReadFrame(0xc4201849c0, 0x56208ec36620, 0xc4202119e0, 0x0, 0x0)
	/usr/share/gocode/src/golang.org/x/net/http2/frame.go:469 +0xa6 fp=0xc420680ef0 sp=0xc420680e18
google.golang.org/grpc/transport.(*framer).readFrame(0xc420211710, 0xc4202119e0, 0xc4202119e0, 0x0, 0x0)
	/usr/share/gocode/src/google.golang.org/grpc/transport/http_util.go:508 +0x31 fp=0xc420680f28 sp=0xc420680ef0
google.golang.org/grpc/transport.(*http2Client).reader(0xc4210ac240)
	/usr/share/gocode/src/google.golang.org/grpc/transport/http2_client.go:937 +0xad fp=0xc420680fd8 sp=0xc420680f28
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420680fe0 sp=0xc420680fd8
created by google.golang.org/grpc/transport.newHTTP2Client
	/usr/share/gocode/src/google.golang.org/grpc/transport/http2_client.go:205 +0x86c

goroutine 190 [chan receive]:
runtime.gopark(0x56208e8625b8, 0xc420e8fcd8, 0x56208e3141f2, 0xc, 0x17, 0x3)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc420d7bd68 sp=0xc420d7bd38
runtime.goparkunlock(0xc420e8fcd8, 0x56208e3141f2, 0xc, 0xc420d7be17, 0x3)
	/usr/lib/golang/src/runtime/proc.go:277 +0x60 fp=0xc420d7bda8 sp=0xc420d7bd68
runtime.chanrecv(0x56208e6c20a0, 0xc420e8fc80, 0xc420d7beb0, 0xffffffffffffff01, 0xc42008fb01)
	/usr/lib/golang/src/runtime/chan.go:513 +0x375 fp=0xc420d7be48 sp=0xc420d7bda8
runtime.chanrecv2(0x56208e6c20a0, 0xc420e8fc80, 0xc420d7beb0, 0x56208db03050)
	/usr/lib/golang/src/runtime/chan.go:400 +0x35 fp=0xc420d7be80 sp=0xc420d7be48
github.com/cockroachdb/cmux.(*muxListener).Accept(0xc420159a00, 0x56208e862008, 0xc437e86500, 0x56208ec3d960, 0xc420177b00)
	<autogenerated>:9 +0x7c fp=0xc420d7bed0 sp=0xc420d7be80
net/http.(*Server).Serve(0xc42012b810, 0x56208ec3c7e0, 0xc420159a00, 0x0, 0x0)
	/usr/lib/golang/src/net/http/server.go:2643 +0x22a fp=0xc420d7bf78 sp=0xc420d7bed0
github.com/coreos/etcd/embed.(*serveCtx).serve.func2(0xc420021140, 0xc42012b810, 0x56208ec3c7e0, 0xc420159a00)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/embed/serve.go:87 +0x41 fp=0xc420d7bfc0 sp=0xc420d7bf78
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420d7bfc8 sp=0xc420d7bfc0
created by github.com/coreos/etcd/embed.(*serveCtx).serve
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/embed/serve.go:87 +0xbc2

goroutine 153 [select, 383 minutes]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc420d80b98 sp=0xc420d80b68
runtime.selectgoImpl(0xc420d80f48, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc420d80e10 sp=0xc420d80b98
runtime.selectgo(0xc420d80f48)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc420d80e38 sp=0xc420d80e10
google.golang.org/grpc/transport.(*http2Client).controller(0xc4210ac240)
	/usr/share/gocode/src/google.golang.org/grpc/transport/http2_client.go:1015 +0x609 fp=0xc420d80fd8 sp=0xc420d80e38
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420d80fe0 sp=0xc420d80fd8
created by google.golang.org/grpc/transport.newHTTP2Client
	/usr/share/gocode/src/google.golang.org/grpc/transport/http2_client.go:235 +0xc2a

goroutine 192 [IO wait, 383 minutes]:
runtime.gopark(0x56208e862580, 0x7fdc68ae2268, 0x56208e30f82a, 0x7, 0x56208db23d1b, 0x5)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc4201127f0 sp=0xc4201127c0
runtime.netpollblock(0x7fdc68ae2240, 0x72, 0xb)
	/usr/lib/golang/src/runtime/netpoll.go:345 +0xc9 fp=0xc420112838 sp=0xc4201127f0
net.runtime_pollWait(0x7fdc68ae2240, 0x72, 0x19)
	/usr/lib/golang/src/runtime/netpoll.go:164 +0x5b fp=0xc420112860 sp=0xc420112838
net.(*pollDesc).wait(0xc4200f1f78, 0x72, 0x56208ec35020, 0x56208ec2ebc0)
	/usr/lib/golang/src/net/fd_poll_runtime.go:75 +0x3a fp=0xc420112888 sp=0xc420112860
net.(*pollDesc).waitRead(0xc4200f1f78, 0xc4210d6000, 0x8000)
	/usr/lib/golang/src/net/fd_poll_runtime.go:80 +0x36 fp=0xc4201128b8 sp=0xc420112888
net.(*netFD).Read(0xc4200f1f10, 0xc4210d6000, 0x8000, 0x8000, 0x0, 0x56208ec35020, 0x56208ec2ebc0)
	/usr/lib/golang/src/net/fd_unix.go:250 +0x1b9 fp=0xc420112920 sp=0xc4201128b8
net.(*conn).Read(0xc42000e220, 0xc4210d6000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/lib/golang/src/net/net.go:181 +0x72 fp=0xc420112990 sp=0xc420112920
github.com/coreos/etcd/pkg/transport.(*limitListenerConn).Read(0xc420177860, 0xc4210d6000, 0x8000, 0x8000, 0x1, 0x56208ec30ba0, 0xc420068040)
	<autogenerated>:16 +0x6d fp=0xc4201129d8 sp=0xc420112990
github.com/cockroachdb/cmux.(*MuxConn).Read(0xc42009cbe0, 0xc4210d6000, 0x8000, 0x8000, 0x9, 0x0, 0x0)
	/usr/share/gocode/src/github.com/cockroachdb/cmux/cmux.go:204 +0x115 fp=0xc420112a38 sp=0xc4201129d8
bufio.(*Reader).Read(0xc420e8fd40, 0xc4200732e8, 0x9, 0x9, 0x9, 0x0, 0x18)
	/usr/lib/golang/src/bufio/bufio.go:213 +0x314 fp=0xc420112ac8 sp=0xc420112a38
io.ReadAtLeast(0x56208ec305e0, 0xc420e8fd40, 0xc4200732e8, 0x9, 0x9, 0x9, 0xc420112b88, 0x56208dabc56a, 0x30)
	/usr/lib/golang/src/io/io.go:307 +0xab fp=0xc420112b28 sp=0xc420112ac8
io.ReadFull(0x56208ec305e0, 0xc420e8fd40, 0xc4200732e8, 0x9, 0x9, 0xa0f9cc0000000000, 0xa000000000000000, 0x2)
	/usr/lib/golang/src/io/io.go:325 +0x5a fp=0xc420112b80 sp=0xc420112b28
golang.org/x/net/http2.readFrameHeader(0xc4200732e8, 0x9, 0x9, 0x56208ec305e0, 0xc420e8fd40, 0x0, 0xc400000000, 0x0, 0x0)
	/usr/share/gocode/src/golang.org/x/net/http2/frame.go:237 +0x7d fp=0xc420112be0 sp=0xc420112b80
golang.org/x/net/http2.(*Framer).ReadFrame(0xc4200732b0, 0x56208ec36620, 0xc420177f80, 0x0, 0x0)
	/usr/share/gocode/src/golang.org/x/net/http2/frame.go:469 +0xa6 fp=0xc420112cb8 sp=0xc420112be0
google.golang.org/grpc/transport.(*framer).readFrame(0xc420177890, 0xc420177f80, 0xc420177f80, 0x0, 0x0)
	/usr/share/gocode/src/google.golang.org/grpc/transport/http_util.go:508 +0x31 fp=0xc420112cf0 sp=0xc420112cb8
google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc4200f5c20, 0xc420177920)
	/usr/share/gocode/src/google.golang.org/grpc/transport/http2_server.go:260 +0x2c0 fp=0xc420112e00 sp=0xc420112cf0
google.golang.org/grpc.(*Server).serveStreams(0xc42012c000, 0x56208ec417e0, 0xc4200f5c20)
	/usr/share/gocode/src/google.golang.org/grpc/server.go:451 +0x154 fp=0xc420112e48 sp=0xc420112e00
google.golang.org/grpc.(*Server).serveNewHTTP2Transport(0xc42012c000, 0x56208ec41a20, 0xc42009cbe0, 0x0, 0x0)
	/usr/share/gocode/src/google.golang.org/grpc/server.go:438 +0x3de fp=0xc420112f10 sp=0xc420112e48
google.golang.org/grpc.(*Server).handleRawConn(0xc42012c000, 0x56208ec41a20, 0xc42009cbe0)
	/usr/share/gocode/src/google.golang.org/grpc/server.go:415 +0x4a3 fp=0xc420112fc8 sp=0xc420112f10
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420112fd0 sp=0xc420112fc8
created by google.golang.org/grpc.(*Server).Serve
	/usr/share/gocode/src/google.golang.org/grpc/server.go:384 +0x580

goroutine 200 [IO wait, 383 minutes]:
runtime.gopark(0x56208e862580, 0x7fdc68ae21a8, 0x56208e30f82a, 0x7, 0x56208db23d1b, 0x5)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc421126ad0 sp=0xc421126aa0
runtime.netpollblock(0x7fdc68ae2180, 0x72, 0xb)
	/usr/lib/golang/src/runtime/netpoll.go:345 +0xc9 fp=0xc421126b18 sp=0xc421126ad0
net.runtime_pollWait(0x7fdc68ae2180, 0x72, 0x1a)
	/usr/lib/golang/src/runtime/netpoll.go:164 +0x5b fp=0xc421126b40 sp=0xc421126b18
net.(*pollDesc).wait(0xc42001eca8, 0x72, 0x56208ec35020, 0x56208ec2ebc0)
	/usr/lib/golang/src/net/fd_poll_runtime.go:75 +0x3a fp=0xc421126b68 sp=0xc421126b40
net.(*pollDesc).waitRead(0xc42001eca8, 0xc42112c000, 0x8000)
	/usr/lib/golang/src/net/fd_poll_runtime.go:80 +0x36 fp=0xc421126b98 sp=0xc421126b68
net.(*netFD).Read(0xc42001ec40, 0xc42112c000, 0x8000, 0x8000, 0x0, 0x56208ec35020, 0x56208ec2ebc0)
	/usr/lib/golang/src/net/fd_unix.go:250 +0x1b9 fp=0xc421126c00 sp=0xc421126b98
net.(*conn).Read(0xc42000e238, 0xc42112c000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/lib/golang/src/net/net.go:181 +0x72 fp=0xc421126c70 sp=0xc421126c00
bufio.(*Reader).Read(0xc4210e8780, 0xc420073968, 0x9, 0x9, 0x9, 0x0, 0x18)
	/usr/lib/golang/src/bufio/bufio.go:213 +0x314 fp=0xc421126d00 sp=0xc421126c70
io.ReadAtLeast(0x56208ec305e0, 0xc4210e8780, 0xc420073968, 0x9, 0x9, 0x9, 0xc421126dc0, 0x56208dabc56a, 0x30)
	/usr/lib/golang/src/io/io.go:307 +0xab fp=0xc421126d60 sp=0xc421126d00
io.ReadFull(0x56208ec305e0, 0xc4210e8780, 0xc420073968, 0x9, 0x9, 0xa0f9cc0000000000, 0xa000000000000000, 0x2)
	/usr/lib/golang/src/io/io.go:325 +0x5a fp=0xc421126db8 sp=0xc421126d60
golang.org/x/net/http2.readFrameHeader(0xc420073968, 0x9, 0x9, 0x56208ec305e0, 0xc4210e8780, 0x0, 0xc400000000, 0x0, 0x0)
	/usr/share/gocode/src/golang.org/x/net/http2/frame.go:237 +0x7d fp=0xc421126e18 sp=0xc421126db8
golang.org/x/net/http2.(*Framer).ReadFrame(0xc420073930, 0x56208ec36620, 0xc421176150, 0x0, 0x0)
	/usr/share/gocode/src/golang.org/x/net/http2/frame.go:469 +0xa6 fp=0xc421126ef0 sp=0xc421126e18
google.golang.org/grpc/transport.(*framer).readFrame(0xc420177d70, 0xc421176150, 0xc421176150, 0x0, 0x0)
	/usr/share/gocode/src/google.golang.org/grpc/transport/http_util.go:508 +0x31 fp=0xc421126f28 sp=0xc421126ef0
google.golang.org/grpc/transport.(*http2Client).reader(0xc4210be480)
	/usr/share/gocode/src/google.golang.org/grpc/transport/http2_client.go:937 +0xad fp=0xc421126fd8 sp=0xc421126f28
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc421126fe0 sp=0xc421126fd8
created by google.golang.org/grpc/transport.newHTTP2Client
	/usr/share/gocode/src/google.golang.org/grpc/transport/http2_client.go:205 +0x86c

goroutine 193 [select, 383 minutes]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc4210b7b80 sp=0xc4210b7b50
runtime.selectgoImpl(0xc4210b7f48, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc4210b7df8 sp=0xc4210b7b80
runtime.selectgo(0xc4210b7f48)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc4210b7e20 sp=0xc4210b7df8
google.golang.org/grpc/transport.(*http2Server).controller(0xc4200f5c20)
	/usr/share/gocode/src/google.golang.org/grpc/transport/http2_server.go:674 +0x736 fp=0xc4210b7fd8 sp=0xc4210b7e20
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc4210b7fe0 sp=0xc4210b7fd8
created by google.golang.org/grpc/transport.newHTTP2Server
	/usr/share/gocode/src/google.golang.org/grpc/transport/http2_server.go:139 +0x6a9

goroutine 198 [IO wait, 383 minutes]:
runtime.gopark(0x56208e862580, 0x7fdc68ae20e8, 0x56208e30f82a, 0x7, 0x56208db23d1b, 0x5)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc421125ad0 sp=0xc421125aa0
runtime.netpollblock(0x7fdc68ae20c0, 0x72, 0xb)
	/usr/lib/golang/src/runtime/netpoll.go:345 +0xc9 fp=0xc421125b18 sp=0xc421125ad0
net.runtime_pollWait(0x7fdc68ae20c0, 0x72, 0x1b)
	/usr/lib/golang/src/runtime/netpoll.go:164 +0x5b fp=0xc421125b40 sp=0xc421125b18
net.(*pollDesc).wait(0xc42001ed18, 0x72, 0x56208ec35020, 0x56208ec2ebc0)
	/usr/lib/golang/src/net/fd_poll_runtime.go:75 +0x3a fp=0xc421125b68 sp=0xc421125b40
net.(*pollDesc).waitRead(0xc42001ed18, 0xc421114000, 0x8000)
	/usr/lib/golang/src/net/fd_poll_runtime.go:80 +0x36 fp=0xc421125b98 sp=0xc421125b68
net.(*netFD).Read(0xc42001ecb0, 0xc421114000, 0x8000, 0x8000, 0x0, 0x56208ec35020, 0x56208ec2ebc0)
	/usr/lib/golang/src/net/fd_unix.go:250 +0x1b9 fp=0xc421125c00 sp=0xc421125b98
net.(*conn).Read(0xc42000e230, 0xc421114000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/lib/golang/src/net/net.go:181 +0x72 fp=0xc421125c70 sp=0xc421125c00
bufio.(*Reader).Read(0xc4210e8540, 0xc420073898, 0x9, 0x9, 0x9, 0x0, 0x18)
	/usr/lib/golang/src/bufio/bufio.go:213 +0x314 fp=0xc421125d00 sp=0xc421125c70
io.ReadAtLeast(0x56208ec305e0, 0xc4210e8540, 0xc420073898, 0x9, 0x9, 0x9, 0xc421125dc0, 0x56208dabc56a, 0x30)
	/usr/lib/golang/src/io/io.go:307 +0xab fp=0xc421125d60 sp=0xc421125d00
io.ReadFull(0x56208ec305e0, 0xc4210e8540, 0xc420073898, 0x9, 0x9, 0xa0f9cc0000000000, 0xa000000000000000, 0x2)
	/usr/lib/golang/src/io/io.go:325 +0x5a fp=0xc421125db8 sp=0xc421125d60
golang.org/x/net/http2.readFrameHeader(0xc420073898, 0x9, 0x9, 0x56208ec305e0, 0xc4210e8540, 0x0, 0xc400000000, 0x0, 0x0)
	/usr/share/gocode/src/golang.org/x/net/http2/frame.go:237 +0x7d fp=0xc421125e18 sp=0xc421125db8
golang.org/x/net/http2.(*Framer).ReadFrame(0xc420073860, 0x56208ec36620, 0xc420177e30, 0x0, 0x0)
	/usr/share/gocode/src/golang.org/x/net/http2/frame.go:469 +0xa6 fp=0xc421125ef0 sp=0xc421125e18
google.golang.org/grpc/transport.(*framer).readFrame(0xc420177c80, 0xc420177e30, 0xc420177e30, 0x0, 0x0)
	/usr/share/gocode/src/google.golang.org/grpc/transport/http_util.go:508 +0x31 fp=0xc421125f28 sp=0xc421125ef0
google.golang.org/grpc/transport.(*http2Client).reader(0xc4210be360)
	/usr/share/gocode/src/google.golang.org/grpc/transport/http2_client.go:937 +0xad fp=0xc421125fd8 sp=0xc421125f28
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc421125fe0 sp=0xc421125fd8
created by google.golang.org/grpc/transport.newHTTP2Client
	/usr/share/gocode/src/google.golang.org/grpc/transport/http2_client.go:205 +0x86c

goroutine 196 [IO wait, 383 minutes]:
runtime.gopark(0x56208e862580, 0x7fdc68ae2028, 0x56208e30f82a, 0x7, 0x56208db23d1b, 0x5)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc420d7ead0 sp=0xc420d7eaa0
runtime.netpollblock(0x7fdc68ae2000, 0x72, 0xb)
	/usr/lib/golang/src/runtime/netpoll.go:345 +0xc9 fp=0xc420d7eb18 sp=0xc420d7ead0
net.runtime_pollWait(0x7fdc68ae2000, 0x72, 0x1c)
	/usr/lib/golang/src/runtime/netpoll.go:164 +0x5b fp=0xc420d7eb40 sp=0xc420d7eb18
net.(*pollDesc).wait(0xc420174618, 0x72, 0x56208ec35020, 0x56208ec2ebc0)
	/usr/lib/golang/src/net/fd_poll_runtime.go:75 +0x3a fp=0xc420d7eb68 sp=0xc420d7eb40
net.(*pollDesc).waitRead(0xc420174618, 0xc421104000, 0x8000)
	/usr/lib/golang/src/net/fd_poll_runtime.go:80 +0x36 fp=0xc420d7eb98 sp=0xc420d7eb68
net.(*netFD).Read(0xc4201745b0, 0xc421104000, 0x8000, 0x8000, 0x0, 0x56208ec35020, 0x56208ec2ebc0)
	/usr/lib/golang/src/net/fd_unix.go:250 +0x1b9 fp=0xc420d7ec00 sp=0xc420d7eb98
net.(*conn).Read(0xc42000e228, 0xc421104000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/lib/golang/src/net/net.go:181 +0x72 fp=0xc420d7ec70 sp=0xc420d7ec00
bufio.(*Reader).Read(0xc4210e8300, 0xc4200737c8, 0x9, 0x9, 0x9, 0x0, 0x18)
	/usr/lib/golang/src/bufio/bufio.go:213 +0x314 fp=0xc420d7ed00 sp=0xc420d7ec70
io.ReadAtLeast(0x56208ec305e0, 0xc4210e8300, 0xc4200737c8, 0x9, 0x9, 0x9, 0xc420d7edc0, 0x56208dabc56a, 0x30)
	/usr/lib/golang/src/io/io.go:307 +0xab fp=0xc420d7ed60 sp=0xc420d7ed00
io.ReadFull(0x56208ec305e0, 0xc4210e8300, 0xc4200737c8, 0x9, 0x9, 0xa0f9cc0000000000, 0xa000000000000000, 0x2)
	/usr/lib/golang/src/io/io.go:325 +0x5a fp=0xc420d7edb8 sp=0xc420d7ed60
golang.org/x/net/http2.readFrameHeader(0xc4200737c8, 0x9, 0x9, 0x56208ec305e0, 0xc4210e8300, 0x0, 0xc400000000, 0x0, 0x0)
	/usr/share/gocode/src/golang.org/x/net/http2/frame.go:237 +0x7d fp=0xc420d7ee18 sp=0xc420d7edb8
golang.org/x/net/http2.(*Framer).ReadFrame(0xc420073790, 0x56208ec36620, 0xc4211761b0, 0x0, 0x0)
	/usr/share/gocode/src/golang.org/x/net/http2/frame.go:469 +0xa6 fp=0xc420d7eef0 sp=0xc420d7ee18
google.golang.org/grpc/transport.(*framer).readFrame(0xc420177b90, 0xc4211761b0, 0xc4211761b0, 0x0, 0x0)
	/usr/share/gocode/src/google.golang.org/grpc/transport/http_util.go:508 +0x31 fp=0xc420d7ef28 sp=0xc420d7eef0
google.golang.org/grpc/transport.(*http2Client).reader(0xc4210be240)
	/usr/share/gocode/src/google.golang.org/grpc/transport/http2_client.go:937 +0xad fp=0xc420d7efd8 sp=0xc420d7ef28
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420d7efe0 sp=0xc420d7efd8
created by google.golang.org/grpc/transport.newHTTP2Client
	/usr/share/gocode/src/google.golang.org/grpc/transport/http2_client.go:205 +0x86c

goroutine 157 [IO wait, 383 minutes]:
runtime.gopark(0x56208e862580, 0x7fdc68b31d68, 0x56208e30f82a, 0x7, 0x56208db23d1b, 0x5)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc4210b5ad0 sp=0xc4210b5aa0
runtime.netpollblock(0x7fdc68b31d40, 0x72, 0xb)
	/usr/lib/golang/src/runtime/netpoll.go:345 +0xc9 fp=0xc4210b5b18 sp=0xc4210b5ad0
net.runtime_pollWait(0x7fdc68b31d40, 0x72, 0x1d)
	/usr/lib/golang/src/runtime/netpoll.go:164 +0x5b fp=0xc4210b5b40 sp=0xc4210b5b18
net.(*pollDesc).wait(0xc42001ed88, 0x72, 0x56208ec35020, 0x56208ec2ebc0)
	/usr/lib/golang/src/net/fd_poll_runtime.go:75 +0x3a fp=0xc4210b5b68 sp=0xc4210b5b40
net.(*pollDesc).waitRead(0xc42001ed88, 0xc4210f4000, 0x8000)
	/usr/lib/golang/src/net/fd_poll_runtime.go:80 +0x36 fp=0xc4210b5b98 sp=0xc4210b5b68
net.(*netFD).Read(0xc42001ed20, 0xc4210f4000, 0x8000, 0x8000, 0x0, 0x56208ec35020, 0x56208ec2ebc0)
	/usr/lib/golang/src/net/fd_unix.go:250 +0x1b9 fp=0xc4210b5c00 sp=0xc4210b5b98
net.(*conn).Read(0xc42007c180, 0xc4210f4000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/lib/golang/src/net/net.go:181 +0x72 fp=0xc4210b5c70 sp=0xc4210b5c00
bufio.(*Reader).Read(0xc4210c07e0, 0xc420184ac8, 0x9, 0x9, 0x9, 0x0, 0x18)
	/usr/lib/golang/src/bufio/bufio.go:213 +0x314 fp=0xc4210b5d00 sp=0xc4210b5c70
io.ReadAtLeast(0x56208ec305e0, 0xc4210c07e0, 0xc420184ac8, 0x9, 0x9, 0x9, 0xc4210b5dc0, 0x56208dabc56a, 0x30)
	/usr/lib/golang/src/io/io.go:307 +0xab fp=0xc4210b5d60 sp=0xc4210b5d00
io.ReadFull(0x56208ec305e0, 0xc4210c07e0, 0xc420184ac8, 0x9, 0x9, 0xa0f9cc0000000000, 0xa000000000000000, 0x2)
	/usr/lib/golang/src/io/io.go:325 +0x5a fp=0xc4210b5db8 sp=0xc4210b5d60
golang.org/x/net/http2.readFrameHeader(0xc420184ac8, 0x9, 0x9, 0x56208ec305e0, 0xc4210c07e0, 0x0, 0xc400000000, 0x0, 0x0)
	/usr/share/gocode/src/golang.org/x/net/http2/frame.go:237 +0x7d fp=0xc4210b5e18 sp=0xc4210b5db8
golang.org/x/net/http2.(*Framer).ReadFrame(0xc420184a90, 0x56208ec36620, 0xc4211760f0, 0x0, 0x0)
	/usr/share/gocode/src/golang.org/x/net/http2/frame.go:469 +0xa6 fp=0xc4210b5ef0 sp=0xc4210b5e18
google.golang.org/grpc/transport.(*framer).readFrame(0xc420211920, 0xc4211760f0, 0xc4211760f0, 0x0, 0x0)
	/usr/share/gocode/src/google.golang.org/grpc/transport/http_util.go:508 +0x31 fp=0xc4210b5f28 sp=0xc4210b5ef0
google.golang.org/grpc/transport.(*http2Client).reader(0xc4210ac360)
	/usr/share/gocode/src/google.golang.org/grpc/transport/http2_client.go:937 +0xad fp=0xc4210b5fd8 sp=0xc4210b5f28
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc4210b5fe0 sp=0xc4210b5fd8
created by google.golang.org/grpc/transport.newHTTP2Client
	/usr/share/gocode/src/google.golang.org/grpc/transport/http2_client.go:205 +0x86c

goroutine 204 [IO wait, 383 minutes]:
runtime.gopark(0x56208e862580, 0x7fdc68b31ca8, 0x56208e30f82a, 0x7, 0x56208db23d1b, 0x5)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc421128ad0 sp=0xc421128aa0
runtime.netpollblock(0x7fdc68b31c80, 0x72, 0xb)
	/usr/lib/golang/src/runtime/netpoll.go:345 +0xc9 fp=0xc421128b18 sp=0xc421128ad0
net.runtime_pollWait(0x7fdc68b31c80, 0x72, 0x1e)
	/usr/lib/golang/src/runtime/netpoll.go:164 +0x5b fp=0xc421128b40 sp=0xc421128b18
net.(*pollDesc).wait(0xc420174688, 0x72, 0x56208ec35020, 0x56208ec2ebc0)
	/usr/lib/golang/src/net/fd_poll_runtime.go:75 +0x3a fp=0xc421128b68 sp=0xc421128b40
net.(*pollDesc).waitRead(0xc420174688, 0xc421188000, 0x8000)
	/usr/lib/golang/src/net/fd_poll_runtime.go:80 +0x36 fp=0xc421128b98 sp=0xc421128b68
net.(*netFD).Read(0xc420174620, 0xc421188000, 0x8000, 0x8000, 0x0, 0x56208ec35020, 0x56208ec2ebc0)
	/usr/lib/golang/src/net/fd_unix.go:250 +0x1b9 fp=0xc421128c00 sp=0xc421128b98
net.(*conn).Read(0xc42000e240, 0xc421188000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/lib/golang/src/net/net.go:181 +0x72 fp=0xc421128c70 sp=0xc421128c00
bufio.(*Reader).Read(0xc4210e8b40, 0xc420073b08, 0x9, 0x9, 0x9, 0x0, 0x18)
	/usr/lib/golang/src/bufio/bufio.go:213 +0x314 fp=0xc421128d00 sp=0xc421128c70
io.ReadAtLeast(0x56208ec305e0, 0xc4210e8b40, 0xc420073b08, 0x9, 0x9, 0x9, 0xc421128dc0, 0x56208dabc56a, 0x30)
	/usr/lib/golang/src/io/io.go:307 +0xab fp=0xc421128d60 sp=0xc421128d00
io.ReadFull(0x56208ec305e0, 0xc4210e8b40, 0xc420073b08, 0x9, 0x9, 0xa0f9cc0000000000, 0xa000000000000000, 0x2)
	/usr/lib/golang/src/io/io.go:325 +0x5a fp=0xc421128db8 sp=0xc421128d60
golang.org/x/net/http2.readFrameHeader(0xc420073b08, 0x9, 0x9, 0x56208ec305e0, 0xc4210e8b40, 0x0, 0xc400000000, 0x0, 0x0)
	/usr/share/gocode/src/golang.org/x/net/http2/frame.go:237 +0x7d fp=0xc421128e18 sp=0xc421128db8
golang.org/x/net/http2.(*Framer).ReadFrame(0xc420073ad0, 0x56208ec36620, 0xc421176270, 0x0, 0x0)
	/usr/share/gocode/src/golang.org/x/net/http2/frame.go:469 +0xa6 fp=0xc421128ef0 sp=0xc421128e18
google.golang.org/grpc/transport.(*framer).readFrame(0xc421176030, 0xc421176270, 0xc421176270, 0x0, 0x0)
	/usr/share/gocode/src/google.golang.org/grpc/transport/http_util.go:508 +0x31 fp=0xc421128f28 sp=0xc421128ef0
google.golang.org/grpc/transport.(*http2Client).reader(0xc4210be5a0)
	/usr/share/gocode/src/google.golang.org/grpc/transport/http2_client.go:937 +0xad fp=0xc421128fd8 sp=0xc421128f28
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc421128fe0 sp=0xc421128fd8
created by google.golang.org/grpc/transport.newHTTP2Client
	/usr/share/gocode/src/google.golang.org/grpc/transport/http2_client.go:205 +0x86c

goroutine 158 [select, 383 minutes]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc4210b4b98 sp=0xc4210b4b68
runtime.selectgoImpl(0xc4210b4f48, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc4210b4e10 sp=0xc4210b4b98
runtime.selectgo(0xc4210b4f48)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc4210b4e38 sp=0xc4210b4e10
google.golang.org/grpc/transport.(*http2Client).controller(0xc4210ac360)
	/usr/share/gocode/src/google.golang.org/grpc/transport/http2_client.go:1015 +0x609 fp=0xc4210b4fd8 sp=0xc4210b4e38
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc4210b4fe0 sp=0xc4210b4fd8
created by google.golang.org/grpc/transport.newHTTP2Client
	/usr/share/gocode/src/google.golang.org/grpc/transport/http2_client.go:235 +0xc2a

goroutine 197 [select, 383 minutes]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc4210b9b98 sp=0xc4210b9b68
runtime.selectgoImpl(0xc4210b9f48, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc4210b9e10 sp=0xc4210b9b98
runtime.selectgo(0xc4210b9f48)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc4210b9e38 sp=0xc4210b9e10
google.golang.org/grpc/transport.(*http2Client).controller(0xc4210be240)
	/usr/share/gocode/src/google.golang.org/grpc/transport/http2_client.go:1015 +0x609 fp=0xc4210b9fd8 sp=0xc4210b9e38
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc4210b9fe0 sp=0xc4210b9fd8
created by google.golang.org/grpc/transport.newHTTP2Client
	/usr/share/gocode/src/google.golang.org/grpc/transport/http2_client.go:235 +0xc2a

goroutine 202 [IO wait, 383 minutes]:
runtime.gopark(0x56208e862580, 0x7fdc68b31be8, 0x56208e30f82a, 0x7, 0x56208db23d1b, 0x5)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc4211297f0 sp=0xc4211297c0
runtime.netpollblock(0x7fdc68b31bc0, 0x72, 0xb)
	/usr/lib/golang/src/runtime/netpoll.go:345 +0xc9 fp=0xc421129838 sp=0xc4211297f0
net.runtime_pollWait(0x7fdc68b31bc0, 0x72, 0x1f)
	/usr/lib/golang/src/runtime/netpoll.go:164 +0x5b fp=0xc421129860 sp=0xc421129838
net.(*pollDesc).wait(0xc42001efb8, 0x72, 0x56208ec35020, 0x56208ec2ebc0)
	/usr/lib/golang/src/net/fd_poll_runtime.go:75 +0x3a fp=0xc421129888 sp=0xc421129860
net.(*pollDesc).waitRead(0xc42001efb8, 0xc42115c000, 0x8000)
	/usr/lib/golang/src/net/fd_poll_runtime.go:80 +0x36 fp=0xc4211298b8 sp=0xc421129888
net.(*netFD).Read(0xc42001ef50, 0xc42115c000, 0x8000, 0x8000, 0x0, 0x56208ec35020, 0x56208ec2ebc0)
	/usr/lib/golang/src/net/fd_unix.go:250 +0x1b9 fp=0xc421129920 sp=0xc4211298b8
net.(*conn).Read(0xc42007c190, 0xc42115c000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/lib/golang/src/net/net.go:181 +0x72 fp=0xc421129990 sp=0xc421129920
github.com/coreos/etcd/pkg/transport.(*limitListenerConn).Read(0xc420211a70, 0xc42115c000, 0x8000, 0x8000, 0x1, 0x56208ec30ba0, 0xc420068040)
	<autogenerated>:16 +0x6d fp=0xc4211299d8 sp=0xc421129990
github.com/cockroachdb/cmux.(*MuxConn).Read(0xc4201c1900, 0xc42115c000, 0x8000, 0x8000, 0x9, 0x0, 0x0)
	/usr/share/gocode/src/github.com/cockroachdb/cmux/cmux.go:204 +0x115 fp=0xc421129a38 sp=0xc4211299d8
bufio.(*Reader).Read(0xc4210e88a0, 0xc420073a38, 0x9, 0x9, 0x9, 0x0, 0x18)
	/usr/lib/golang/src/bufio/bufio.go:213 +0x314 fp=0xc421129ac8 sp=0xc421129a38
io.ReadAtLeast(0x56208ec305e0, 0xc4210e88a0, 0xc420073a38, 0x9, 0x9, 0x9, 0xc421129b88, 0x56208dabc56a, 0x30)
	/usr/lib/golang/src/io/io.go:307 +0xab fp=0xc421129b28 sp=0xc421129ac8
io.ReadFull(0x56208ec305e0, 0xc4210e88a0, 0xc420073a38, 0x9, 0x9, 0xa0f9cc0000000000, 0xa000000000000000, 0x2)
	/usr/lib/golang/src/io/io.go:325 +0x5a fp=0xc421129b80 sp=0xc421129b28
golang.org/x/net/http2.readFrameHeader(0xc420073a38, 0x9, 0x9, 0x56208ec305e0, 0xc4210e88a0, 0x0, 0xc400000000, 0x0, 0x0)
	/usr/share/gocode/src/golang.org/x/net/http2/frame.go:237 +0x7d fp=0xc421129be0 sp=0xc421129b80
golang.org/x/net/http2.(*Framer).ReadFrame(0xc420073a00, 0x56208ec36620, 0xc421176210, 0x0, 0x0)
	/usr/share/gocode/src/golang.org/x/net/http2/frame.go:469 +0xa6 fp=0xc421129cb8 sp=0xc421129be0
google.golang.org/grpc/transport.(*framer).readFrame(0xc420177e90, 0xc421176210, 0xc421176210, 0x0, 0x0)
	/usr/share/gocode/src/google.golang.org/grpc/transport/http_util.go:508 +0x31 fp=0xc421129cf0 sp=0xc421129cb8
google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc4200f5e60, 0xc420177f20)
	/usr/share/gocode/src/google.golang.org/grpc/transport/http2_server.go:260 +0x2c0 fp=0xc421129e00 sp=0xc421129cf0
google.golang.org/grpc.(*Server).serveStreams(0xc42012c000, 0x56208ec417e0, 0xc4200f5e60)
	/usr/share/gocode/src/google.golang.org/grpc/server.go:451 +0x154 fp=0xc421129e48 sp=0xc421129e00
google.golang.org/grpc.(*Server).serveNewHTTP2Transport(0xc42012c000, 0x56208ec41a20, 0xc4201c1900, 0x0, 0x0)
	/usr/share/gocode/src/google.golang.org/grpc/server.go:438 +0x3de fp=0xc421129f10 sp=0xc421129e48
google.golang.org/grpc.(*Server).handleRawConn(0xc42012c000, 0x56208ec41a20, 0xc4201c1900)
	/usr/share/gocode/src/google.golang.org/grpc/server.go:415 +0x4a3 fp=0xc421129fc8 sp=0xc421129f10
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc421129fd0 sp=0xc421129fc8
created by google.golang.org/grpc.(*Server).Serve
	/usr/share/gocode/src/google.golang.org/grpc/server.go:384 +0x580

goroutine 212 [IO wait, 383 minutes]:
runtime.gopark(0x56208e862580, 0x7fdc68b31b28, 0x56208e30f82a, 0x7, 0x56208db23d1b, 0x5)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc42112a7f0 sp=0xc42112a7c0
runtime.netpollblock(0x7fdc68b31b00, 0x72, 0xb)
	/usr/lib/golang/src/runtime/netpoll.go:345 +0xc9 fp=0xc42112a838 sp=0xc42112a7f0
net.runtime_pollWait(0x7fdc68b31b00, 0x72, 0x20)
	/usr/lib/golang/src/runtime/netpoll.go:164 +0x5b fp=0xc42112a860 sp=0xc42112a838
net.(*pollDesc).wait(0xc42001f028, 0x72, 0x56208ec35020, 0x56208ec2ebc0)
	/usr/lib/golang/src/net/fd_poll_runtime.go:75 +0x3a fp=0xc42112a888 sp=0xc42112a860
net.(*pollDesc).waitRead(0xc42001f028, 0xc42113c000, 0x8000)
	/usr/lib/golang/src/net/fd_poll_runtime.go:80 +0x36 fp=0xc42112a8b8 sp=0xc42112a888
net.(*netFD).Read(0xc42001efc0, 0xc42113c000, 0x8000, 0x8000, 0x0, 0x56208ec35020, 0x56208ec2ebc0)
	/usr/lib/golang/src/net/fd_unix.go:250 +0x1b9 fp=0xc42112a920 sp=0xc42112a8b8
net.(*conn).Read(0xc42007c1e0, 0xc42113c000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/lib/golang/src/net/net.go:181 +0x72 fp=0xc42112a990 sp=0xc42112a920
github.com/coreos/etcd/pkg/transport.(*limitListenerConn).Read(0xc420211b00, 0xc42113c000, 0x8000, 0x8000, 0x1, 0x56208ec30ba0, 0xc420068040)
	<autogenerated>:16 +0x6d fp=0xc42112a9d8 sp=0xc42112a990
github.com/cockroachdb/cmux.(*MuxConn).Read(0xc4201c19a0, 0xc42113c000, 0x8000, 0x8000, 0x9, 0x0, 0x0)
	/usr/share/gocode/src/github.com/cockroachdb/cmux/cmux.go:204 +0x115 fp=0xc42112aa38 sp=0xc42112a9d8
bufio.(*Reader).Read(0xc4210c0900, 0xc420184b98, 0x9, 0x9, 0x9, 0x0, 0x18)
	/usr/lib/golang/src/bufio/bufio.go:213 +0x314 fp=0xc42112aac8 sp=0xc42112aa38
io.ReadAtLeast(0x56208ec305e0, 0xc4210c0900, 0xc420184b98, 0x9, 0x9, 0x9, 0xc42112ab88, 0x56208dabc56a, 0x30)
	/usr/lib/golang/src/io/io.go:307 +0xab fp=0xc42112ab28 sp=0xc42112aac8
io.ReadFull(0x56208ec305e0, 0xc4210c0900, 0xc420184b98, 0x9, 0x9, 0xa0f9cc0000000000, 0xa000000000000000, 0x2)
	/usr/lib/golang/src/io/io.go:325 +0x5a fp=0xc42112ab80 sp=0xc42112ab28
golang.org/x/net/http2.readFrameHeader(0xc420184b98, 0x9, 0x9, 0x56208ec305e0, 0xc4210c0900, 0x0, 0xc400000000, 0x0, 0x0)
	/usr/share/gocode/src/golang.org/x/net/http2/frame.go:237 +0x7d fp=0xc42112abe0 sp=0xc42112ab80
golang.org/x/net/http2.(*Framer).ReadFrame(0xc420184b60, 0x56208ec36620, 0xc420177e60, 0x0, 0x0)
	/usr/share/gocode/src/golang.org/x/net/http2/frame.go:469 +0xa6 fp=0xc42112acb8 sp=0xc42112abe0
google.golang.org/grpc/transport.(*framer).readFrame(0xc420211ce0, 0xc420177e60, 0xc420177e60, 0x0, 0x0)
	/usr/share/gocode/src/google.golang.org/grpc/transport/http_util.go:508 +0x31 fp=0xc42112acf0 sp=0xc42112acb8
google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc420022990, 0xc420211d70)
	/usr/share/gocode/src/google.golang.org/grpc/transport/http2_server.go:260 +0x2c0 fp=0xc42112ae00 sp=0xc42112acf0
google.golang.org/grpc.(*Server).serveStreams(0xc42012c000, 0x56208ec417e0, 0xc420022990)
	/usr/share/gocode/src/google.golang.org/grpc/server.go:451 +0x154 fp=0xc42112ae48 sp=0xc42112ae00
google.golang.org/grpc.(*Server).serveNewHTTP2Transport(0xc42012c000, 0x56208ec41a20, 0xc4201c19a0, 0x0, 0x0)
	/usr/share/gocode/src/google.golang.org/grpc/server.go:438 +0x3de fp=0xc42112af10 sp=0xc42112ae48
google.golang.org/grpc.(*Server).handleRawConn(0xc42012c000, 0x56208ec41a20, 0xc4201c19a0)
	/usr/share/gocode/src/google.golang.org/grpc/server.go:415 +0x4a3 fp=0xc42112afc8 sp=0xc42112af10
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc42112afd0 sp=0xc42112afc8
created by google.golang.org/grpc.(*Server).Serve
	/usr/share/gocode/src/google.golang.org/grpc/server.go:384 +0x580

goroutine 214 [IO wait, 383 minutes]:
runtime.gopark(0x56208e862580, 0x7fdc68b31a68, 0x56208e30f82a, 0x7, 0x56208db23d1b, 0x5)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc4210b27f0 sp=0xc4210b27c0
runtime.netpollblock(0x7fdc68b31a40, 0x72, 0xb)
	/usr/lib/golang/src/runtime/netpoll.go:345 +0xc9 fp=0xc4210b2838 sp=0xc4210b27f0
net.runtime_pollWait(0x7fdc68b31a40, 0x72, 0x21)
	/usr/lib/golang/src/runtime/netpoll.go:164 +0x5b fp=0xc4210b2860 sp=0xc4210b2838
net.(*pollDesc).wait(0xc42001f098, 0x72, 0x56208ec35020, 0x56208ec2ebc0)
	/usr/lib/golang/src/net/fd_poll_runtime.go:75 +0x3a fp=0xc4210b2888 sp=0xc4210b2860
net.(*pollDesc).waitRead(0xc42001f098, 0xc42114c000, 0x8000)
	/usr/lib/golang/src/net/fd_poll_runtime.go:80 +0x36 fp=0xc4210b28b8 sp=0xc4210b2888
net.(*netFD).Read(0xc42001f030, 0xc42114c000, 0x8000, 0x8000, 0x0, 0x56208ec35020, 0x56208ec2ebc0)
	/usr/lib/golang/src/net/fd_unix.go:250 +0x1b9 fp=0xc4210b2920 sp=0xc4210b28b8
net.(*conn).Read(0xc42007c1f8, 0xc42114c000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/lib/golang/src/net/net.go:181 +0x72 fp=0xc4210b2990 sp=0xc4210b2920
github.com/coreos/etcd/pkg/transport.(*limitListenerConn).Read(0xc420211b90, 0xc42114c000, 0x8000, 0x8000, 0x1, 0x56208ec30ba0, 0xc420068040)
	<autogenerated>:16 +0x6d fp=0xc4210b29d8 sp=0xc4210b2990
github.com/cockroachdb/cmux.(*MuxConn).Read(0xc4201c1a40, 0xc42114c000, 0x8000, 0x8000, 0x9, 0x0, 0x0)
	/usr/share/gocode/src/github.com/cockroachdb/cmux/cmux.go:204 +0x115 fp=0xc4210b2a38 sp=0xc4210b29d8
bufio.(*Reader).Read(0xc4210c0a80, 0xc420184c68, 0x9, 0x9, 0x9, 0x0, 0x18)
	/usr/lib/golang/src/bufio/bufio.go:213 +0x314 fp=0xc4210b2ac8 sp=0xc4210b2a38
io.ReadAtLeast(0x56208ec305e0, 0xc4210c0a80, 0xc420184c68, 0x9, 0x9, 0x9, 0xc4210b2b88, 0x56208dabc56a, 0x30)
	/usr/lib/golang/src/io/io.go:307 +0xab fp=0xc4210b2b28 sp=0xc4210b2ac8
io.ReadFull(0x56208ec305e0, 0xc4210c0a80, 0xc420184c68, 0x9, 0x9, 0xa0f9cc0000000000, 0xa000000000000000, 0x2)
	/usr/lib/golang/src/io/io.go:325 +0x5a fp=0xc4210b2b80 sp=0xc4210b2b28
golang.org/x/net/http2.readFrameHeader(0xc420184c68, 0x9, 0x9, 0x56208ec305e0, 0xc4210c0a80, 0x0, 0xc400000000, 0x0, 0x0)
	/usr/share/gocode/src/golang.org/x/net/http2/frame.go:237 +0x7d fp=0xc4210b2be0 sp=0xc4210b2b80
golang.org/x/net/http2.(*Framer).ReadFrame(0xc420184c30, 0x56208ec36620, 0xc4211761e0, 0x0, 0x0)
	/usr/share/gocode/src/golang.org/x/net/http2/frame.go:469 +0xa6 fp=0xc4210b2cb8 sp=0xc4210b2be0
google.golang.org/grpc/transport.(*framer).readFrame(0xc420211dd0, 0xc4211761e0, 0xc4211761e0, 0x0, 0x0)
	/usr/share/gocode/src/google.golang.org/grpc/transport/http_util.go:508 +0x31 fp=0xc4210b2cf0 sp=0xc4210b2cb8
google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc420022a20, 0xc420211e60)
	/usr/share/gocode/src/google.golang.org/grpc/transport/http2_server.go:260 +0x2c0 fp=0xc4210b2e00 sp=0xc4210b2cf0
google.golang.org/grpc.(*Server).serveStreams(0xc42012c000, 0x56208ec417e0, 0xc420022a20)
	/usr/share/gocode/src/google.golang.org/grpc/server.go:451 +0x154 fp=0xc4210b2e48 sp=0xc4210b2e00
google.golang.org/grpc.(*Server).serveNewHTTP2Transport(0xc42012c000, 0x56208ec41a20, 0xc4201c1a40, 0x0, 0x0)
	/usr/share/gocode/src/google.golang.org/grpc/server.go:438 +0x3de fp=0xc4210b2f10 sp=0xc4210b2e48
google.golang.org/grpc.(*Server).handleRawConn(0xc42012c000, 0x56208ec41a20, 0xc4201c1a40)
	/usr/share/gocode/src/google.golang.org/grpc/server.go:415 +0x4a3 fp=0xc4210b2fc8 sp=0xc4210b2f10
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc4210b2fd0 sp=0xc4210b2fc8
created by google.golang.org/grpc.(*Server).Serve
	/usr/share/gocode/src/google.golang.org/grpc/server.go:384 +0x580

goroutine 216 [IO wait, 383 minutes]:
runtime.gopark(0x56208e862580, 0x7fdc68b319a8, 0x56208e30f82a, 0x7, 0x56208db23d1b, 0x5)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc4211747f0 sp=0xc4211747c0
runtime.netpollblock(0x7fdc68b31980, 0x72, 0xb)
	/usr/lib/golang/src/runtime/netpoll.go:345 +0xc9 fp=0xc421174838 sp=0xc4211747f0
net.runtime_pollWait(0x7fdc68b31980, 0x72, 0x22)
	/usr/lib/golang/src/runtime/netpoll.go:164 +0x5b fp=0xc421174860 sp=0xc421174838
net.(*pollDesc).wait(0xc42001f178, 0x72, 0x56208ec35020, 0x56208ec2ebc0)
	/usr/lib/golang/src/net/fd_poll_runtime.go:75 +0x3a fp=0xc421174888 sp=0xc421174860
net.(*pollDesc).waitRead(0xc42001f178, 0xc421178000, 0x8000)
	/usr/lib/golang/src/net/fd_poll_runtime.go:80 +0x36 fp=0xc4211748b8 sp=0xc421174888
net.(*netFD).Read(0xc42001f110, 0xc421178000, 0x8000, 0x8000, 0x0, 0x56208ec35020, 0x56208ec2ebc0)
	/usr/lib/golang/src/net/fd_unix.go:250 +0x1b9 fp=0xc421174920 sp=0xc4211748b8
net.(*conn).Read(0xc42007c220, 0xc421178000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/lib/golang/src/net/net.go:181 +0x72 fp=0xc421174990 sp=0xc421174920
github.com/coreos/etcd/pkg/transport.(*limitListenerConn).Read(0xc420211c20, 0xc421178000, 0x8000, 0x8000, 0x1, 0x56208ec30ba0, 0xc420068040)
	<autogenerated>:16 +0x6d fp=0xc4211749d8 sp=0xc421174990
github.com/cockroachdb/cmux.(*MuxConn).Read(0xc4201c1ae0, 0xc421178000, 0x8000, 0x8000, 0x9, 0x0, 0x0)
	/usr/share/gocode/src/github.com/cockroachdb/cmux/cmux.go:204 +0x115 fp=0xc421174a38 sp=0xc4211749d8
bufio.(*Reader).Read(0xc4210c0c00, 0xc420184d38, 0x9, 0x9, 0x9, 0x0, 0x18)
	/usr/lib/golang/src/bufio/bufio.go:213 +0x314 fp=0xc421174ac8 sp=0xc421174a38
io.ReadAtLeast(0x56208ec305e0, 0xc4210c0c00, 0xc420184d38, 0x9, 0x9, 0x9, 0xc421174b88, 0x56208dabc56a, 0x30)
	/usr/lib/golang/src/io/io.go:307 +0xab fp=0xc421174b28 sp=0xc421174ac8
io.ReadFull(0x56208ec305e0, 0xc4210c0c00, 0xc420184d38, 0x9, 0x9, 0xa0f9cc0000000000, 0xa000000000000000, 0x2)
	/usr/lib/golang/src/io/io.go:325 +0x5a fp=0xc421174b80 sp=0xc421174b28
golang.org/x/net/http2.readFrameHeader(0xc420184d38, 0x9, 0x9, 0x56208ec305e0, 0xc4210c0c00, 0x0, 0xc400000000, 0x0, 0x0)
	/usr/share/gocode/src/golang.org/x/net/http2/frame.go:237 +0x7d fp=0xc421174be0 sp=0xc421174b80
golang.org/x/net/http2.(*Framer).ReadFrame(0xc420184d00, 0x56208ec36620, 0xc4211a80c0, 0x0, 0x0)
	/usr/share/gocode/src/golang.org/x/net/http2/frame.go:469 +0xa6 fp=0xc421174cb8 sp=0xc421174be0
google.golang.org/grpc/transport.(*framer).readFrame(0xc420211ec0, 0xc4211a80c0, 0xc4211a80c0, 0x0, 0x0)
	/usr/share/gocode/src/google.golang.org/grpc/transport/http_util.go:508 +0x31 fp=0xc421174cf0 sp=0xc421174cb8
google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc420022ab0, 0xc420211f50)
	/usr/share/gocode/src/google.golang.org/grpc/transport/http2_server.go:260 +0x2c0 fp=0xc421174e00 sp=0xc421174cf0
google.golang.org/grpc.(*Server).serveStreams(0xc42012c000, 0x56208ec417e0, 0xc420022ab0)
	/usr/share/gocode/src/google.golang.org/grpc/server.go:451 +0x154 fp=0xc421174e48 sp=0xc421174e00
google.golang.org/grpc.(*Server).serveNewHTTP2Transport(0xc42012c000, 0x56208ec41a20, 0xc4201c1ae0, 0x0, 0x0)
	/usr/share/gocode/src/google.golang.org/grpc/server.go:438 +0x3de fp=0xc421174f10 sp=0xc421174e48
google.golang.org/grpc.(*Server).handleRawConn(0xc42012c000, 0x56208ec41a20, 0xc4201c1ae0)
	/usr/share/gocode/src/google.golang.org/grpc/server.go:415 +0x4a3 fp=0xc421174fc8 sp=0xc421174f10
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc421174fd0 sp=0xc421174fc8
created by google.golang.org/grpc.(*Server).Serve
	/usr/share/gocode/src/google.golang.org/grpc/server.go:384 +0x580

goroutine 218 [IO wait, 383 minutes]:
runtime.gopark(0x56208e862580, 0x7fdc68b318e8, 0x56208e30f82a, 0x7, 0x56208db23d1b, 0x5)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc4210b67f0 sp=0xc4210b67c0
runtime.netpollblock(0x7fdc68b318c0, 0x72, 0xb)
	/usr/lib/golang/src/runtime/netpoll.go:345 +0xc9 fp=0xc4210b6838 sp=0xc4210b67f0
net.runtime_pollWait(0x7fdc68b318c0, 0x72, 0x23)
	/usr/lib/golang/src/runtime/netpoll.go:164 +0x5b fp=0xc4210b6860 sp=0xc4210b6838
net.(*pollDesc).wait(0xc42001f1e8, 0x72, 0x56208ec35020, 0x56208ec2ebc0)
	/usr/lib/golang/src/net/fd_poll_runtime.go:75 +0x3a fp=0xc4210b6888 sp=0xc4210b6860
net.(*pollDesc).waitRead(0xc42001f1e8, 0xc421198000, 0x8000)
	/usr/lib/golang/src/net/fd_poll_runtime.go:80 +0x36 fp=0xc4210b68b8 sp=0xc4210b6888
net.(*netFD).Read(0xc42001f180, 0xc421198000, 0x8000, 0x8000, 0x0, 0x56208ec35020, 0x56208ec2ebc0)
	/usr/lib/golang/src/net/fd_unix.go:250 +0x1b9 fp=0xc4210b6920 sp=0xc4210b68b8
net.(*conn).Read(0xc42007c228, 0xc421198000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/lib/golang/src/net/net.go:181 +0x72 fp=0xc4210b6990 sp=0xc4210b6920
github.com/coreos/etcd/pkg/transport.(*limitListenerConn).Read(0xc420211cb0, 0xc421198000, 0x8000, 0x8000, 0x1, 0x56208ec30ba0, 0xc420068040)
	<autogenerated>:16 +0x6d fp=0xc4210b69d8 sp=0xc4210b6990
github.com/cockroachdb/cmux.(*MuxConn).Read(0xc4201c1860, 0xc421198000, 0x8000, 0x8000, 0x9, 0x0, 0x0)
	/usr/share/gocode/src/github.com/cockroachdb/cmux/cmux.go:204 +0x115 fp=0xc4210b6a38 sp=0xc4210b69d8
bufio.(*Reader).Read(0xc4210c0d80, 0xc420184e08, 0x9, 0x9, 0x9, 0x0, 0x18)
	/usr/lib/golang/src/bufio/bufio.go:213 +0x314 fp=0xc4210b6ac8 sp=0xc4210b6a38
io.ReadAtLeast(0x56208ec305e0, 0xc4210c0d80, 0xc420184e08, 0x9, 0x9, 0x9, 0xc4210b6b88, 0x56208dabc56a, 0x30)
	/usr/lib/golang/src/io/io.go:307 +0xab fp=0xc4210b6b28 sp=0xc4210b6ac8
io.ReadFull(0x56208ec305e0, 0xc4210c0d80, 0xc420184e08, 0x9, 0x9, 0xa0f9cc0000000000, 0xa000000000000000, 0x2)
	/usr/lib/golang/src/io/io.go:325 +0x5a fp=0xc4210b6b80 sp=0xc4210b6b28
golang.org/x/net/http2.readFrameHeader(0xc420184e08, 0x9, 0x9, 0x56208ec305e0, 0xc4210c0d80, 0x0, 0xc400000000, 0x0, 0x0)
	/usr/share/gocode/src/golang.org/x/net/http2/frame.go:237 +0x7d fp=0xc4210b6be0 sp=0xc4210b6b80
golang.org/x/net/http2.(*Framer).ReadFrame(0xc420184dd0, 0x56208ec36620, 0xc4211762a0, 0x0, 0x0)
	/usr/share/gocode/src/golang.org/x/net/http2/frame.go:469 +0xa6 fp=0xc4210b6cb8 sp=0xc4210b6be0
google.golang.org/grpc/transport.(*framer).readFrame(0xc420211fb0, 0xc4211762a0, 0xc4211762a0, 0x0, 0x0)
	/usr/share/gocode/src/google.golang.org/grpc/transport/http_util.go:508 +0x31 fp=0xc4210b6cf0 sp=0xc4210b6cb8
google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc420022b40, 0xc4211a8060)
	/usr/share/gocode/src/google.golang.org/grpc/transport/http2_server.go:260 +0x2c0 fp=0xc4210b6e00 sp=0xc4210b6cf0
google.golang.org/grpc.(*Server).serveStreams(0xc42012c000, 0x56208ec417e0, 0xc420022b40)
	/usr/share/gocode/src/google.golang.org/grpc/server.go:451 +0x154 fp=0xc4210b6e48 sp=0xc4210b6e00
google.golang.org/grpc.(*Server).serveNewHTTP2Transport(0xc42012c000, 0x56208ec41a20, 0xc4201c1860, 0x0, 0x0)
	/usr/share/gocode/src/google.golang.org/grpc/server.go:438 +0x3de fp=0xc4210b6f10 sp=0xc4210b6e48
google.golang.org/grpc.(*Server).handleRawConn(0xc42012c000, 0x56208ec41a20, 0xc4201c1860)
	/usr/share/gocode/src/google.golang.org/grpc/server.go:415 +0x4a3 fp=0xc4210b6fc8 sp=0xc4210b6f10
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc4210b6fd0 sp=0xc4210b6fc8
created by google.golang.org/grpc.(*Server).Serve
	/usr/share/gocode/src/google.golang.org/grpc/server.go:384 +0x580

goroutine 199 [select, 383 minutes]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc4210b3b98 sp=0xc4210b3b68
runtime.selectgoImpl(0xc4210b3f48, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc4210b3e10 sp=0xc4210b3b98
runtime.selectgo(0xc4210b3f48)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc4210b3e38 sp=0xc4210b3e10
google.golang.org/grpc/transport.(*http2Client).controller(0xc4210be360)
	/usr/share/gocode/src/google.golang.org/grpc/transport/http2_client.go:1015 +0x609 fp=0xc4210b3fd8 sp=0xc4210b3e38
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc4210b3fe0 sp=0xc4210b3fd8
created by google.golang.org/grpc/transport.newHTTP2Client
	/usr/share/gocode/src/google.golang.org/grpc/transport/http2_client.go:235 +0xc2a

goroutine 213 [select, 383 minutes]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc42112bb80 sp=0xc42112bb50
runtime.selectgoImpl(0xc42112bf48, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc42112bdf8 sp=0xc42112bb80
runtime.selectgo(0xc42112bf48)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc42112be20 sp=0xc42112bdf8
google.golang.org/grpc/transport.(*http2Server).controller(0xc420022990)
	/usr/share/gocode/src/google.golang.org/grpc/transport/http2_server.go:674 +0x736 fp=0xc42112bfd8 sp=0xc42112be20
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc42112bfe0 sp=0xc42112bfd8
created by google.golang.org/grpc/transport.newHTTP2Server
	/usr/share/gocode/src/google.golang.org/grpc/transport/http2_server.go:139 +0x6a9

goroutine 201 [select, 383 minutes]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc420d6cb98 sp=0xc420d6cb68
runtime.selectgoImpl(0xc420d6cf48, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc420d6ce10 sp=0xc420d6cb98
runtime.selectgo(0xc420d6cf48)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc420d6ce38 sp=0xc420d6ce10
google.golang.org/grpc/transport.(*http2Client).controller(0xc4210be480)
	/usr/share/gocode/src/google.golang.org/grpc/transport/http2_client.go:1015 +0x609 fp=0xc420d6cfd8 sp=0xc420d6ce38
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420d6cfe0 sp=0xc420d6cfd8
created by google.golang.org/grpc/transport.newHTTP2Client
	/usr/share/gocode/src/google.golang.org/grpc/transport/http2_client.go:235 +0xc2a

goroutine 215 [select, 383 minutes]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc421173b80 sp=0xc421173b50
runtime.selectgoImpl(0xc421173f48, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc421173df8 sp=0xc421173b80
runtime.selectgo(0xc421173f48)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc421173e20 sp=0xc421173df8
google.golang.org/grpc/transport.(*http2Server).controller(0xc420022a20)
	/usr/share/gocode/src/google.golang.org/grpc/transport/http2_server.go:674 +0x736 fp=0xc421173fd8 sp=0xc421173e20
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc421173fe0 sp=0xc421173fd8
created by google.golang.org/grpc/transport.newHTTP2Server
	/usr/share/gocode/src/google.golang.org/grpc/transport/http2_server.go:139 +0x6a9

goroutine 203 [select, 383 minutes]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc421127b80 sp=0xc421127b50
runtime.selectgoImpl(0xc421127f48, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc421127df8 sp=0xc421127b80
runtime.selectgo(0xc421127f48)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc421127e20 sp=0xc421127df8
google.golang.org/grpc/transport.(*http2Server).controller(0xc4200f5e60)
	/usr/share/gocode/src/google.golang.org/grpc/transport/http2_server.go:674 +0x736 fp=0xc421127fd8 sp=0xc421127e20
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc421127fe0 sp=0xc421127fd8
created by google.golang.org/grpc/transport.newHTTP2Server
	/usr/share/gocode/src/google.golang.org/grpc/transport/http2_server.go:139 +0x6a9

goroutine 217 [select, 383 minutes]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc421175b80 sp=0xc421175b50
runtime.selectgoImpl(0xc421175f48, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc421175df8 sp=0xc421175b80
runtime.selectgo(0xc421175f48)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc421175e20 sp=0xc421175df8
google.golang.org/grpc/transport.(*http2Server).controller(0xc420022ab0)
	/usr/share/gocode/src/google.golang.org/grpc/transport/http2_server.go:674 +0x736 fp=0xc421175fd8 sp=0xc421175e20
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc421175fe0 sp=0xc421175fd8
created by google.golang.org/grpc/transport.newHTTP2Server
	/usr/share/gocode/src/google.golang.org/grpc/transport/http2_server.go:139 +0x6a9

goroutine 205 [select, 383 minutes]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc4210b8b98 sp=0xc4210b8b68
runtime.selectgoImpl(0xc4210b8f48, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc4210b8e10 sp=0xc4210b8b98
runtime.selectgo(0xc4210b8f48)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc4210b8e38 sp=0xc4210b8e10
google.golang.org/grpc/transport.(*http2Client).controller(0xc4210be5a0)
	/usr/share/gocode/src/google.golang.org/grpc/transport/http2_client.go:1015 +0x609 fp=0xc4210b8fd8 sp=0xc4210b8e38
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc4210b8fe0 sp=0xc4210b8fd8
created by google.golang.org/grpc/transport.newHTTP2Client
	/usr/share/gocode/src/google.golang.org/grpc/transport/http2_client.go:235 +0xc2a

goroutine 219 [select, 383 minutes]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc421124b80 sp=0xc421124b50
runtime.selectgoImpl(0xc421124f48, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc421124df8 sp=0xc421124b80
runtime.selectgo(0xc421124f48)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc421124e20 sp=0xc421124df8
google.golang.org/grpc/transport.(*http2Server).controller(0xc420022b40)
	/usr/share/gocode/src/google.golang.org/grpc/transport/http2_server.go:674 +0x736 fp=0xc421124fd8 sp=0xc421124e20
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc421124fe0 sp=0xc421124fd8
created by google.golang.org/grpc/transport.newHTTP2Server
	/usr/share/gocode/src/google.golang.org/grpc/transport/http2_server.go:139 +0x6a9

goroutine 504043 [chan receive, 3 minutes]:
runtime.gopark(0x56208e8625b8, 0xc4338764d8, 0x56208e3141f2, 0xc, 0xc475cd5217, 0x3)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc4d9b669d8 sp=0xc4d9b669a8
runtime.goparkunlock(0xc4338764d8, 0x56208e3141f2, 0xc, 0x180017, 0x3)
	/usr/lib/golang/src/runtime/proc.go:277 +0x60 fp=0xc4d9b66a18 sp=0xc4d9b669d8
runtime.chanrecv(0x56208e6c13e0, 0xc433876480, 0x0, 0x1, 0x56208ec32e20)
	/usr/lib/golang/src/runtime/chan.go:513 +0x375 fp=0xc4d9b66ab8 sp=0xc4d9b66a18
runtime.chanrecv1(0x56208e6c13e0, 0xc433876480, 0x0)
	/usr/lib/golang/src/runtime/chan.go:395 +0x35 fp=0xc4d9b66af0 sp=0xc4d9b66ab8
github.com/coreos/etcd/rafthttp.(*streamHandler).ServeHTTP(0xc420071480, 0x56208ec3c520, 0xc4368f3ea0, 0xc438e5fc00)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/http.go:323 +0xa41 fp=0xc4d9b66c90 sp=0xc4d9b66af0
net/http.(*ServeMux).ServeHTTP(0xc420182360, 0x56208ec3c520, 0xc4368f3ea0, 0xc438e5fc00)
	/usr/lib/golang/src/net/http/server.go:2238 +0x132 fp=0xc4d9b66cd0 sp=0xc4d9b66c90
net/http.(*ServeMux).ServeHTTP(0xc4201823c0, 0x56208ec3c520, 0xc4368f3ea0, 0xc438e5fc00)
	/usr/lib/golang/src/net/http/server.go:2238 +0x132 fp=0xc4d9b66d10 sp=0xc4d9b66cd0
net/http.serverHandler.ServeHTTP(0xc42012ae70, 0x56208ec3c520, 0xc4368f3ea0, 0xc438e5fc00)
	/usr/lib/golang/src/net/http/server.go:2568 +0x94 fp=0xc4d9b66d58 sp=0xc4d9b66d10
net/http.(*conn).serve(0xc432dfc280, 0x56208ec3d8a0, 0xc475cd4fc0)
	/usr/lib/golang/src/net/http/server.go:1825 +0x614 fp=0xc4d9b66fc8 sp=0xc4d9b66d58
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc4d9b66fd0 sp=0xc4d9b66fc8
created by net/http.(*Server).Serve
	/usr/lib/golang/src/net/http/server.go:2668 +0x2d0

goroutine 503998 [chan receive, 3 minutes]:
runtime.gopark(0x56208e8625b8, 0xc432df4958, 0x56208e3141f2, 0xc, 0xc475cd4317, 0x3)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc488b269d8 sp=0xc488b269a8
runtime.goparkunlock(0xc432df4958, 0x56208e3141f2, 0xc, 0x180017, 0x3)
	/usr/lib/golang/src/runtime/proc.go:277 +0x60 fp=0xc488b26a18 sp=0xc488b269d8
runtime.chanrecv(0x56208e6c13e0, 0xc432df4900, 0x0, 0x1, 0x56208ec32e20)
	/usr/lib/golang/src/runtime/chan.go:513 +0x375 fp=0xc488b26ab8 sp=0xc488b26a18
runtime.chanrecv1(0x56208e6c13e0, 0xc432df4900, 0x0)
	/usr/lib/golang/src/runtime/chan.go:395 +0x35 fp=0xc488b26af0 sp=0xc488b26ab8
github.com/coreos/etcd/rafthttp.(*streamHandler).ServeHTTP(0xc420071480, 0x56208ec3c520, 0xc4368f3a40, 0xc438e5f400)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/http.go:323 +0xa41 fp=0xc488b26c90 sp=0xc488b26af0
net/http.(*ServeMux).ServeHTTP(0xc420182360, 0x56208ec3c520, 0xc4368f3a40, 0xc438e5f400)
	/usr/lib/golang/src/net/http/server.go:2238 +0x132 fp=0xc488b26cd0 sp=0xc488b26c90
net/http.(*ServeMux).ServeHTTP(0xc4201823c0, 0x56208ec3c520, 0xc4368f3a40, 0xc438e5f400)
	/usr/lib/golang/src/net/http/server.go:2238 +0x132 fp=0xc488b26d10 sp=0xc488b26cd0
net/http.serverHandler.ServeHTTP(0xc42012ae70, 0x56208ec3c520, 0xc4368f3a40, 0xc438e5f400)
	/usr/lib/golang/src/net/http/server.go:2568 +0x94 fp=0xc488b26d58 sp=0xc488b26d10
net/http.(*conn).serve(0xc433453c20, 0x56208ec3d8a0, 0xc475cd4180)
	/usr/lib/golang/src/net/http/server.go:1825 +0x614 fp=0xc488b26fc8 sp=0xc488b26d58
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc488b26fd0 sp=0xc488b26fc8
created by net/http.(*Server).Serve
	/usr/lib/golang/src/net/http/server.go:2668 +0x2d0

goroutine 367191 [select, 107 minutes]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc420d688f0 sp=0xc420d688c0
runtime.selectgoImpl(0xc420d68ee0, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc420d68b68 sp=0xc420d688f0
runtime.selectgo(0xc420d68ee0)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc420d68b90 sp=0xc420d68b68
net/http.(*persistConn).readLoop(0xc438094c60)
	/usr/lib/golang/src/net/http/transport.go:1599 +0x9ee fp=0xc420d68fd8 sp=0xc420d68b90
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420d68fe0 sp=0xc420d68fd8
created by net/http.(*Transport).dialConn
	/usr/lib/golang/src/net/http/transport.go:1117 +0xa37

goroutine 504107 [select]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc475a33be8 sp=0xc475a33bb8
runtime.selectgoImpl(0xc475a33f48, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc475a33e60 sp=0xc475a33be8
runtime.selectgo(0xc475a33f48)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc475a33e88 sp=0xc475a33e60
net/http.(*persistConn).writeLoop(0xc432b39680)
	/usr/lib/golang/src/net/http/transport.go:1704 +0x43c fp=0xc475a33fd8 sp=0xc475a33e88
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc475a33fe0 sp=0xc475a33fd8
created by net/http.(*Transport).dialConn
	/usr/lib/golang/src/net/http/transport.go:1118 +0xa5c

goroutine 290410 [semacquire]:
runtime.gopark(0x56208e8625b8, 0xc47a1d9218, 0x56208e312e73, 0xa, 0xc47910091a, 0x3)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc420d82c30 sp=0xc420d82c00
runtime.goparkunlock(0xc47a1d9218, 0x56208e312e73, 0xa, 0x1a, 0x3)
	/usr/lib/golang/src/runtime/proc.go:277 +0x60 fp=0xc420d82c70 sp=0xc420d82c30
sync.runtime_notifyListWait(0xc47a1d9210, 0x562000000958)
	/usr/lib/golang/src/runtime/sema.go:297 +0x10f fp=0xc420d82cc0 sp=0xc420d82c70
sync.(*Cond).Wait(0xc47a1d9200)
	/usr/lib/golang/src/sync/cond.go:57 +0x8b fp=0xc420d82cf0 sp=0xc420d82cc0
net/http.(*connReader).abortPendingRead(0xc4752bc8c0)
	/usr/lib/golang/src/net/http/server.go:686 +0xc1 fp=0xc420d82d30 sp=0xc420d82cf0
net/http.(*response).finishRequest(0xc486a381c0)
	/usr/lib/golang/src/net/http/server.go:1543 +0xa6 fp=0xc420d82d58 sp=0xc420d82d30
net/http.(*conn).serve(0xc478fb45a0, 0x56208ec3d8a0, 0xc4752bc880)
	/usr/lib/golang/src/net/http/server.go:1830 +0x654 fp=0xc420d82fc8 sp=0xc420d82d58
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420d82fd0 sp=0xc420d82fc8
created by net/http.(*Server).Serve
	/usr/lib/golang/src/net/http/server.go:2668 +0x2d0

goroutine 290827 [select, 159 minutes]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc488ac28f0 sp=0xc488ac28c0
runtime.selectgoImpl(0xc488ac2ee0, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc488ac2b68 sp=0xc488ac28f0
runtime.selectgo(0xc488ac2ee0)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc488ac2b90 sp=0xc488ac2b68
net/http.(*persistConn).readLoop(0xc488d47b00)
	/usr/lib/golang/src/net/http/transport.go:1599 +0x9ee fp=0xc488ac2fd8 sp=0xc488ac2b90
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc488ac2fe0 sp=0xc488ac2fd8
created by net/http.(*Transport).dialConn
	/usr/lib/golang/src/net/http/transport.go:1117 +0xa37

goroutine 290828 [select, 159 minutes]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc488b22be8 sp=0xc488b22bb8
runtime.selectgoImpl(0xc488b22f48, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc488b22e60 sp=0xc488b22be8
runtime.selectgo(0xc488b22f48)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc488b22e88 sp=0xc488b22e60
net/http.(*persistConn).writeLoop(0xc488d47b00)
	/usr/lib/golang/src/net/http/transport.go:1704 +0x43c fp=0xc488b22fd8 sp=0xc488b22e88
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc488b22fe0 sp=0xc488b22fd8
created by net/http.(*Transport).dialConn
	/usr/lib/golang/src/net/http/transport.go:1118 +0xa5c

goroutine 483649 [IO wait]:
runtime.gopark(0x56208e862580, 0x7fd9dea6ab58, 0x56208e30f82a, 0x7, 0x56208db23d1b, 0x5)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc43a568838 sp=0xc43a568808
runtime.netpollblock(0x7fd9dea6ab30, 0x72, 0xb)
	/usr/lib/golang/src/runtime/netpoll.go:345 +0xc9 fp=0xc43a568880 sp=0xc43a568838
net.runtime_pollWait(0x7fd9dea6ab30, 0x72, 0xc)
	/usr/lib/golang/src/runtime/netpoll.go:164 +0x5b fp=0xc43a5688a8 sp=0xc43a568880
net.(*pollDesc).wait(0xc437cef9c8, 0x72, 0x56208ec35020, 0x56208ec2ebc0)
	/usr/lib/golang/src/net/fd_poll_runtime.go:75 +0x3a fp=0xc43a5688d0 sp=0xc43a5688a8
net.(*pollDesc).waitRead(0xc437cef9c8, 0xc439b8f000, 0x1000)
	/usr/lib/golang/src/net/fd_poll_runtime.go:80 +0x36 fp=0xc43a568900 sp=0xc43a5688d0
net.(*netFD).Read(0xc437cef960, 0xc439b8f000, 0x1000, 0x1000, 0x0, 0x56208ec35020, 0x56208ec2ebc0)
	/usr/lib/golang/src/net/fd_unix.go:250 +0x1b9 fp=0xc43a568968 sp=0xc43a568900
net.(*conn).Read(0xc43905efc0, 0xc439b8f000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/lib/golang/src/net/net.go:181 +0x72 fp=0xc43a5689d8 sp=0xc43a568968
github.com/coreos/etcd/pkg/transport.timeoutConn.Read(0x56208ec41ea0, 0xc43905efc0, 0x0, 0x0, 0xc439b8f000, 0x1000, 0x1000, 0x4, 0xc43a568a90, 0x56208dab0ac6)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/pkg/transport/timeout_conn.go:43 +0x5f fp=0xc43a568a20 sp=0xc43a5689d8
github.com/coreos/etcd/pkg/transport.(*timeoutConn).Read(0xc4368afce0, 0xc439b8f000, 0x1000, 0x1000, 0xc439b8d380, 0xc43d046e10, 0xc43a568be5)
	<autogenerated>:42 +0x88 fp=0xc43a568a80 sp=0xc43a568a20
net/http.(*persistConn).Read(0xc439b8a240, 0xc439b8f000, 0x1000, 0x1000, 0x4, 0x56208db00940, 0xc439b8d380)
	/usr/lib/golang/src/net/http/transport.go:1316 +0x14d fp=0xc43a568af8 sp=0xc43a568a80
bufio.(*Reader).fill(0xc439b8d140)
	/usr/lib/golang/src/bufio/bufio.go:97 +0x119 fp=0xc43a568b58 sp=0xc43a568af8
bufio.(*Reader).Peek(0xc439b8d140, 0x1, 0xc43a568be5, 0x1, 0x0, 0xc477d99260, 0x0)
	/usr/lib/golang/src/bufio/bufio.go:129 +0x69 fp=0xc43a568b90 sp=0xc43a568b58
net/http.(*persistConn).readLoop(0xc439b8a240)
	/usr/lib/golang/src/net/http/transport.go:1474 +0x198 fp=0xc43a568fd8 sp=0xc43a568b90
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc43a568fe0 sp=0xc43a568fd8
created by net/http.(*Transport).dialConn
	/usr/lib/golang/src/net/http/transport.go:1117 +0xa37

goroutine 507139 [IO wait]:
runtime.gopark(0x56208e862580, 0x7fd9dea69058, 0x56208e30f82a, 0x7, 0x56208db23d1b, 0x5)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc48928b530 sp=0xc48928b500
runtime.netpollblock(0x7fd9dea69030, 0x72, 0xb)
	/usr/lib/golang/src/runtime/netpoll.go:345 +0xc9 fp=0xc48928b578 sp=0xc48928b530
net.runtime_pollWait(0x7fd9dea69030, 0x72, 0x8)
	/usr/lib/golang/src/runtime/netpoll.go:164 +0x5b fp=0xc48928b5a0 sp=0xc48928b578
net.(*pollDesc).wait(0xc47d70cbc8, 0x72, 0x56208ec35020, 0x56208ec2ebc0)
	/usr/lib/golang/src/net/fd_poll_runtime.go:75 +0x3a fp=0xc48928b5c8 sp=0xc48928b5a0
net.(*pollDesc).waitRead(0xc47d70cbc8, 0xc4380963d1, 0x1)
	/usr/lib/golang/src/net/fd_poll_runtime.go:80 +0x36 fp=0xc48928b5f8 sp=0xc48928b5c8
net.(*netFD).Read(0xc47d70cb60, 0xc4380963d1, 0x1, 0x1, 0x0, 0x56208ec35020, 0x56208ec2ebc0)
	/usr/lib/golang/src/net/fd_unix.go:250 +0x1b9 fp=0xc48928b660 sp=0xc48928b5f8
net.(*conn).Read(0xc47966f678, 0xc4380963d1, 0x1, 0x1, 0x0, 0x0, 0x0)
	/usr/lib/golang/src/net/net.go:181 +0x72 fp=0xc48928b6d0 sp=0xc48928b660
github.com/coreos/etcd/pkg/transport.timeoutConn.Read(0x56208ec41ea0, 0xc47966f678, 0x12a05f200, 0x12a05f200, 0xc4380963d1, 0x1, 0x1, 0xc48928b768, 0x56208db19be7, 0xc438096410)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/pkg/transport/timeout_conn.go:43 +0x5f fp=0xc48928b718 sp=0xc48928b6d0
github.com/coreos/etcd/pkg/transport.(*timeoutConn).Read(0xc4380ae7a0, 0xc4380963d1, 0x1, 0x1, 0x0, 0x0, 0x0)
	<autogenerated>:42 +0x88 fp=0xc48928b778 sp=0xc48928b718
net/http.(*connReader).backgroundRead(0xc4380963c0)
	/usr/lib/golang/src/net/http/server.go:656 +0x5a fp=0xc48928b7d8 sp=0xc48928b778
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc48928b7e0 sp=0xc48928b7d8
created by net/http.(*connReader).startBackgroundRead
	/usr/lib/golang/src/net/http/server.go:652 +0xe1

goroutine 507140 [IO wait]:
runtime.gopark(0x56208e862580, 0x7fdc68ae24a8, 0x56208e30f82a, 0x7, 0x56208db23d1b, 0x5)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc43671b530 sp=0xc43671b500
runtime.netpollblock(0x7fdc68ae2480, 0x72, 0xb)
	/usr/lib/golang/src/runtime/netpoll.go:345 +0xc9 fp=0xc43671b578 sp=0xc43671b530
net.runtime_pollWait(0x7fdc68ae2480, 0x72, 0x16)
	/usr/lib/golang/src/runtime/netpoll.go:164 +0x5b fp=0xc43671b5a0 sp=0xc43671b578
net.(*pollDesc).wait(0xc4200f1db8, 0x72, 0x56208ec35020, 0x56208ec2ebc0)
	/usr/lib/golang/src/net/fd_poll_runtime.go:75 +0x3a fp=0xc43671b5c8 sp=0xc43671b5a0
net.(*pollDesc).waitRead(0xc4200f1db8, 0xc4201f3f11, 0x1)
	/usr/lib/golang/src/net/fd_poll_runtime.go:80 +0x36 fp=0xc43671b5f8 sp=0xc43671b5c8
net.(*netFD).Read(0xc4200f1d50, 0xc4201f3f11, 0x1, 0x1, 0x0, 0x56208ec35020, 0x56208ec2ebc0)
	/usr/lib/golang/src/net/fd_unix.go:250 +0x1b9 fp=0xc43671b660 sp=0xc43671b5f8
net.(*conn).Read(0xc42000e048, 0xc4201f3f11, 0x1, 0x1, 0x0, 0x0, 0x0)
	/usr/lib/golang/src/net/net.go:181 +0x72 fp=0xc43671b6d0 sp=0xc43671b660
github.com/coreos/etcd/pkg/transport.timeoutConn.Read(0x56208ec41ea0, 0xc42000e048, 0x12a05f200, 0x12a05f200, 0xc4201f3f11, 0x1, 0x1, 0xc43671b768, 0x56208db19be7, 0xc4201f3f90)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/pkg/transport/timeout_conn.go:43 +0x5f fp=0xc43671b718 sp=0xc43671b6d0
github.com/coreos/etcd/pkg/transport.(*timeoutConn).Read(0xc420158ca0, 0xc4201f3f11, 0x1, 0x1, 0x0, 0x0, 0x0)
	<autogenerated>:42 +0x88 fp=0xc43671b778 sp=0xc43671b718
net/http.(*connReader).backgroundRead(0xc4201f3f00)
	/usr/lib/golang/src/net/http/server.go:656 +0x5a fp=0xc43671b7d8 sp=0xc43671b778
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc43671b7e0 sp=0xc43671b7d8
created by net/http.(*connReader).startBackgroundRead
	/usr/lib/golang/src/net/http/server.go:652 +0xe1

goroutine 504108 [chan receive, 3 minutes]:
runtime.gopark(0x56208e8625b8, 0xc47d0825f8, 0x56208e3141f2, 0xc, 0xc47bd39a17, 0x3)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc475a2b9d8 sp=0xc475a2b9a8
runtime.goparkunlock(0xc47d0825f8, 0x56208e3141f2, 0xc, 0x180017, 0x3)
	/usr/lib/golang/src/runtime/proc.go:277 +0x60 fp=0xc475a2ba18 sp=0xc475a2b9d8
runtime.chanrecv(0x56208e6c13e0, 0xc47d0825a0, 0x0, 0x1, 0x56208ec32e20)
	/usr/lib/golang/src/runtime/chan.go:513 +0x375 fp=0xc475a2bab8 sp=0xc475a2ba18
runtime.chanrecv1(0x56208e6c13e0, 0xc47d0825a0, 0x0)
	/usr/lib/golang/src/runtime/chan.go:395 +0x35 fp=0xc475a2baf0 sp=0xc475a2bab8
github.com/coreos/etcd/rafthttp.(*streamHandler).ServeHTTP(0xc420071480, 0x56208ec3c520, 0xc4775f2a80, 0xc436865400)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/http.go:323 +0xa41 fp=0xc475a2bc90 sp=0xc475a2baf0
net/http.(*ServeMux).ServeHTTP(0xc420182360, 0x56208ec3c520, 0xc4775f2a80, 0xc436865400)
	/usr/lib/golang/src/net/http/server.go:2238 +0x132 fp=0xc475a2bcd0 sp=0xc475a2bc90
net/http.(*ServeMux).ServeHTTP(0xc4201823c0, 0x56208ec3c520, 0xc4775f2a80, 0xc436865400)
	/usr/lib/golang/src/net/http/server.go:2238 +0x132 fp=0xc475a2bd10 sp=0xc475a2bcd0
net/http.serverHandler.ServeHTTP(0xc42012ae70, 0x56208ec3c520, 0xc4775f2a80, 0xc436865400)
	/usr/lib/golang/src/net/http/server.go:2568 +0x94 fp=0xc475a2bd58 sp=0xc475a2bd10
net/http.(*conn).serve(0xc432dfd040, 0x56208ec3d8a0, 0xc47bd39840)
	/usr/lib/golang/src/net/http/server.go:1825 +0x614 fp=0xc475a2bfc8 sp=0xc475a2bd58
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc475a2bfd0 sp=0xc475a2bfc8
created by net/http.(*Server).Serve
	/usr/lib/golang/src/net/http/server.go:2668 +0x2d0

goroutine 483650 [select]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc476711be8 sp=0xc476711bb8
runtime.selectgoImpl(0xc476711f48, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc476711e60 sp=0xc476711be8
runtime.selectgo(0xc476711f48)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc476711e88 sp=0xc476711e60
net/http.(*persistConn).writeLoop(0xc439b8a240)
	/usr/lib/golang/src/net/http/transport.go:1704 +0x43c fp=0xc476711fd8 sp=0xc476711e88
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc476711fe0 sp=0xc476711fd8
created by net/http.(*Transport).dialConn
	/usr/lib/golang/src/net/http/transport.go:1118 +0xa5c

goroutine 367192 [select, 107 minutes]:
runtime.gopark(0x56208e8625f0, 0x0, 0x56208e30f2bb, 0x6, 0x18, 0x2)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc475a2abe8 sp=0xc475a2abb8
runtime.selectgoImpl(0xc475a2af48, 0x0, 0x18)
	/usr/lib/golang/src/runtime/select.go:423 +0x1368 fp=0xc475a2ae60 sp=0xc475a2abe8
runtime.selectgo(0xc475a2af48)
	/usr/lib/golang/src/runtime/select.go:238 +0x1c fp=0xc475a2ae88 sp=0xc475a2ae60
net/http.(*persistConn).writeLoop(0xc438094c60)
	/usr/lib/golang/src/net/http/transport.go:1704 +0x43c fp=0xc475a2afd8 sp=0xc475a2ae88
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc475a2afe0 sp=0xc475a2afd8
created by net/http.(*Transport).dialConn
	/usr/lib/golang/src/net/http/transport.go:1118 +0xa5c

goroutine 367162 [semacquire]:
runtime.gopark(0x56208e8625b8, 0xc438096418, 0x56208e312e73, 0xa, 0xc47d70cb1a, 0x3)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc4796f8c30 sp=0xc4796f8c00
runtime.goparkunlock(0xc438096418, 0x56208e312e73, 0xa, 0x1a, 0x3)
	/usr/lib/golang/src/runtime/proc.go:277 +0x60 fp=0xc4796f8c70 sp=0xc4796f8c30
sync.runtime_notifyListWait(0xc438096410, 0x56200000061b)
	/usr/lib/golang/src/runtime/sema.go:297 +0x10f fp=0xc4796f8cc0 sp=0xc4796f8c70
sync.(*Cond).Wait(0xc438096400)
	/usr/lib/golang/src/sync/cond.go:57 +0x8b fp=0xc4796f8cf0 sp=0xc4796f8cc0
net/http.(*connReader).abortPendingRead(0xc4380963c0)
	/usr/lib/golang/src/net/http/server.go:686 +0xc1 fp=0xc4796f8d30 sp=0xc4796f8cf0
net/http.(*response).finishRequest(0xc486a382a0)
	/usr/lib/golang/src/net/http/server.go:1543 +0xa6 fp=0xc4796f8d58 sp=0xc4796f8d30
net/http.(*conn).serve(0xc4791d7360, 0x56208ec3d8a0, 0xc438096380)
	/usr/lib/golang/src/net/http/server.go:1830 +0x654 fp=0xc4796f8fc8 sp=0xc4796f8d58
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc4796f8fd0 sp=0xc4796f8fc8
created by net/http.(*Server).Serve
	/usr/lib/golang/src/net/http/server.go:2668 +0x2d0

goroutine 504149 [chan receive, 3 minutes]:
runtime.gopark(0x56208e8625b8, 0xc436817018, 0x56208e3141f2, 0xc, 0xc488654417, 0x3)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc4d9b6c9d8 sp=0xc4d9b6c9a8
runtime.goparkunlock(0xc436817018, 0x56208e3141f2, 0xc, 0x180017, 0x3)
	/usr/lib/golang/src/runtime/proc.go:277 +0x60 fp=0xc4d9b6ca18 sp=0xc4d9b6c9d8
runtime.chanrecv(0x56208e6c13e0, 0xc436816fc0, 0x0, 0x1, 0x56208ec32e20)
	/usr/lib/golang/src/runtime/chan.go:513 +0x375 fp=0xc4d9b6cab8 sp=0xc4d9b6ca18
runtime.chanrecv1(0x56208e6c13e0, 0xc436816fc0, 0x0)
	/usr/lib/golang/src/runtime/chan.go:395 +0x35 fp=0xc4d9b6caf0 sp=0xc4d9b6cab8
github.com/coreos/etcd/rafthttp.(*streamHandler).ServeHTTP(0xc420071480, 0x56208ec3c520, 0xc4775f2e00, 0xc436865900)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/rafthttp/http.go:323 +0xa41 fp=0xc4d9b6cc90 sp=0xc4d9b6caf0
net/http.(*ServeMux).ServeHTTP(0xc420182360, 0x56208ec3c520, 0xc4775f2e00, 0xc436865900)
	/usr/lib/golang/src/net/http/server.go:2238 +0x132 fp=0xc4d9b6ccd0 sp=0xc4d9b6cc90
net/http.(*ServeMux).ServeHTTP(0xc4201823c0, 0x56208ec3c520, 0xc4775f2e00, 0xc436865900)
	/usr/lib/golang/src/net/http/server.go:2238 +0x132 fp=0xc4d9b6cd10 sp=0xc4d9b6ccd0
net/http.serverHandler.ServeHTTP(0xc42012ae70, 0x56208ec3c520, 0xc4775f2e00, 0xc436865900)
	/usr/lib/golang/src/net/http/server.go:2568 +0x94 fp=0xc4d9b6cd58 sp=0xc4d9b6cd10
net/http.(*conn).serve(0xc432dfd400, 0x56208ec3d8a0, 0xc488654240)
	/usr/lib/golang/src/net/http/server.go:1825 +0x614 fp=0xc4d9b6cfc8 sp=0xc4d9b6cd58
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc4d9b6cfd0 sp=0xc4d9b6cfc8
created by net/http.(*Server).Serve
	/usr/lib/golang/src/net/http/server.go:2668 +0x2d0

goroutine 507138 [IO wait]:
runtime.gopark(0x56208e862580, 0x7fd9def4b098, 0x56208e30f82a, 0x7, 0x56208db23d1b, 0x5)
	/usr/lib/golang/src/runtime/proc.go:271 +0x140 fp=0xc43685a530 sp=0xc43685a500
runtime.netpollblock(0x7fd9def4b070, 0x72, 0xb)
	/usr/lib/golang/src/runtime/netpoll.go:345 +0xc9 fp=0xc43685a578 sp=0xc43685a530
net.runtime_pollWait(0x7fd9def4b070, 0x72, 0x8a)
	/usr/lib/golang/src/runtime/netpoll.go:164 +0x5b fp=0xc43685a5a0 sp=0xc43685a578
net.(*pollDesc).wait(0xc479100998, 0x72, 0x56208ec35020, 0x56208ec2ebc0)
	/usr/lib/golang/src/net/fd_poll_runtime.go:75 +0x3a fp=0xc43685a5c8 sp=0xc43685a5a0
net.(*pollDesc).waitRead(0xc479100998, 0xc4752bc8d1, 0x1)
	/usr/lib/golang/src/net/fd_poll_runtime.go:80 +0x36 fp=0xc43685a5f8 sp=0xc43685a5c8
net.(*netFD).Read(0xc479100930, 0xc4752bc8d1, 0x1, 0x1, 0x0, 0x56208ec35020, 0x56208ec2ebc0)
	/usr/lib/golang/src/net/fd_unix.go:250 +0x1b9 fp=0xc43685a660 sp=0xc43685a5f8
net.(*conn).Read(0xc42000e328, 0xc4752bc8d1, 0x1, 0x1, 0x0, 0x0, 0x0)
	/usr/lib/golang/src/net/net.go:181 +0x72 fp=0xc43685a6d0 sp=0xc43685a660
github.com/coreos/etcd/pkg/transport.timeoutConn.Read(0x56208ec41ea0, 0xc42000e328, 0x12a05f200, 0x12a05f200, 0xc4752bc8d1, 0x1, 0x1, 0xc43685a768, 0x56208db19be7, 0xc47a1d9210)
	/builddir/build/BUILD/etcd-21fdcc6443f3267111051240f0eca839acc96a39/src/github.com/coreos/etcd/pkg/transport/timeout_conn.go:43 +0x5f fp=0xc43685a718 sp=0xc43685a6d0
github.com/coreos/etcd/pkg/transport.(*timeoutConn).Read(0xc479033ac0, 0xc4752bc8d1, 0x1, 0x1, 0x0, 0x0, 0x0)
	<autogenerated>:42 +0x88 fp=0xc43685a778 sp=0xc43685a718
net/http.(*connReader).backgroundRead(0xc4752bc8c0)
	/usr/lib/golang/src/net/http/server.go:656 +0x5a fp=0xc43685a7d8 sp=0xc43685a778
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc43685a7e0 sp=0xc43685a7d8
created by net/http.(*connReader).startBackgroundRead
	/usr/lib/golang/src/net/http/server.go:652 +0xe1
./start-etcd.sh: line 10:  1457 Aborted                 (core dumped) etcd --name node2 --listen-peer-urls http://10.167.133.32:2380 --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://10.167.133.32:2379 --initial-advertise-peer-urls http://10.167.133.32:2380 --initial-cluster-token etcd-cluster-1 --initial-cluster node1=http://10.167.133.31:2380,master=http://10.167.133.28:2380,node2=http://10.167.133.32:2380 --data-dir /home/node-2/etcd/data --initial-cluster-state new

@WanLinghao
Copy link
Author

etcd start up command:

etcd --name node2 \
--listen-peer-urls http://10.167.133.32:2380 \
--listen-client-urls http://0.0.0.0:2379 \
--advertise-client-urls http://10.167.133.32:2379 \
--initial-advertise-peer-urls http://10.167.133.32:2380 \
--initial-cluster-token etcd-cluster-1 \
--initial-cluster node1=http://10.167.133.31:2380,master=http://10.167.133.28:2380,node2=http://10.167.133.32:2380 \
--data-dir /home/node-2/etcd/data \
--initial-cluster-state new

@xiang90
Copy link
Contributor

xiang90 commented Jan 9, 2018

fatal error: runtime: out of memory

well... it runs out of memory. so you need to provide more memory i guess.

please share your scripts to run the test if you want anyone to look into it more.

@WanLinghao
Copy link
Author

@xiang90 hello,we want to confirm the count of pod. What 5Mi pods mean

@xiang90
Copy link
Contributor

xiang90 commented Jan 9, 2018

it means 5,000,000 kvs.

@WanLinghao
Copy link
Author

insert_pod.sh:

#!/bin/bash
source "./get_json.sh"

insert_pod(){
        endpoint=$1
        key_file=$2
        total_count=$3
        count=0
        failure=0
        dup=0
        echo create $total_count key-value!
        while [ $count -lt $total_count ]
        do
                echo current key:$count,current failure times:$failure,current duplicate times:$dup
                get_pod_key
                get_pod_json
                key=$pod_key
                value=$pod_json
                etcdctl --endpoints=$endpoint get $key&>/dev/null
                if [ $? -eq 0 ]; then
                        #etcd has duplicate key
                        echo duplicate key!
                        dup=$[dup+1]
                        continue
                fi
                etcdctl --endpoints=$endpoint set $key $value&>/dev/null
                if [ $? -ne 0 ]; then
                        #etcdctl set key failed 
                        echo set key failure!
                        failure=$[failure+1]
                        continue
                fi
                echo $key>>$key_file
                count=$[count+1]
        done
        return $failure
}

insert_audit.sh:

#!/bin/bash
source "./get_json.sh"
insert_audit(){
        endpoint=$1
        key_file=$2
        total_count=$3
        count=0
        failure=0
        dup=0
        echo create $total_count key-value!
        while [ $count -lt $total_count ]
        do
                echo current key:$count,current failure times:$failure,current duplicate times:$dup
                get_audit_key
                get_audit_json
                key=$audit_key
                value=$audit_json
                etcdctl --endpoints=$endpoint get $key&>/dev/null
                if [ $? -eq 0 ]; then
                        #etcd has duplicate key
                        echo duplicate key!
                        dup=$[dup+1]
                        continue
                fi
                etcdctl --endpoints=$endpoint set $key $value&>/dev/null
                if [ $? -ne 0 ]; then
                        #etcdctl set key failed 
                        echo set key failure!
                        failure=$[failure+1]
                        continue
                fi
                echo $key>>$key_file
                count=$[count+1]
        done
        return $failure
}

get_json.sh:

pod_json_part1='"apiVersion":"v1","items":[{"apiVersion":"v1","kind":"Pod","metadata":{"creationTimestamp":"2018-01-02T03:22:32Z","name":"'

pod_json_part2='","namespace":"default","resourceVersion":"458","selfLink":"/api/v1/namespaces/default/pods/alpine","uid":"'
pod_json_part3='"},"spec":{"containers":[{"image":"ubuntu","imagePullPolicy":"Always","name":"alpine","resources":{},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","volumeMounts":[{"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount","name":"default-token-5mcf5","readOnly":true}]}],"dnsPolicy":"ClusterFirst","nodeName":"127.0.0.1","restartPolicy":"Always","schedulerName":"default-scheduler","securityContext":{},"serviceAccount":"default","serviceAccountName":"default","terminationGracePeriodSeconds":30,"tolerations":[{"effect":"NoExecute","key":"node.kubernetes.io/not-ready","operator":"Exists","tolerationSeconds":300},{"effect":"NoExecute","key":"node.kubernetes.io/unreachable","operator":"Exists","tolerationSeconds":300}],"volumes":[{"name":"default-token-5mcf5","secret":{"defaultMode":420,"secretName":"default-token-5mcf5"}}]},"status":{"conditions":[{"lastProbeTime":null,"lastTransitionTime":"2018-01-02T03:22:32Z","status":"True","type":"Initialized"},{"lastProbeTime":null,"lastTransitionTime":"2018-01-02T03:22:32Z","message":"containerswithunreadystatus:[alpine]","reason":"ContainersNotReady","status":"False","type":"Ready"},{"lastProbeTime":null,"lastTransitionTime":"2018-01-02T03:22:32Z","status":"True","type":"PodScheduled"}],"containerStatuses":[{"containerID":"docker://b417e5b36d3d0e7cdefd0757df467441e37700fec59cea2a74610826774bc9e5","image":"ubuntu:latest","imageID":"docker-pullable://ubuntu@sha256:7ab3fe2a5b60ff63da8d428ca69aa4abf09a8cf6c51f9abd4f0e589d59b12eb3","lastState":{"terminated":{"containerID":"docker://3c270c60b091e2fb331b3cab64491e953dc356b9981c6cc2c89484e9fdba39dc","exitCode":0,"finishedAt":"2018-01-02T03:24:33Z","reason":"Completed","startedAt":"2018-01-02T03:24:33Z"}},"name":"alpine","ready":false,"restartCount":5,"state":{"terminated":{"containerID":"docker://b417e5b36d3d0e7cdefd0757df467441e37700fec59cea2a74610826774bc9e5","exitCode":0,"finishedAt":"2018-01-02T03:26:03Z","reason":"Completed","startedAt":"2018-01-02T03:26:03Z"}}}],"hostIP":"127.0.0.1","phase":"Running","podIP":"172.17.0.3","qosClass":"BestEffort","startTime":"2018-01-02T03:22:32Z"}}],"kind":"List","metadata":{"resourceVersion":"","selfLink":""}}'
audit_json_part1='{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-01-02T02:46:14Z"},"level":"Metadata","timestamp":"2018-01-02T02:46:14Z","auditID":"'
audit_json_part2='","stage":"RequestReceived","requestURI":"/apis/rbac.authorization.k8s.io/v1/rolebindings?limit=500\u0026resourceVersion=0","verb":"list","user":{"username":"system:apiserver","uid":"23c10956-3479-41af-af1f-958f4081e351","groups":["system:masters"]},"sourceIPs":["127.0.0.1"],"objectRef":{"resource":"rolebindings","apiGroup":"rbac.authorization.k8s.io","apiVersion":"v1"},"requestReceivedTimestamp":"2018-01-02T02:46:14.439794Z","stageTimestamp":"2018-01-02T02:46:14.439794Z"}'
get_random_len_random_str(){
        num=$(shuf -i 5-15 -n 1)
        random_len_str=$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w ${num:-32} | head -n 1)    
}

get_fix_len_random_str(){
        fix_len_str=$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w ${1:-32} | head -n 1)    
}

get_pod_json(){
        get_random_len_random_str
        pod_name=$random_len_str
        get_fix_len_random_str 36
        pod_uid=$fix_len_str

        pod_json=$(echo $pod_json_part1$pod_name$pod_json_part2$pod_uid$pod_json_part3)
}

get_audit_json(){
        get_fix_len_random_str 36
        audit_uid=$fix_len_str
        audit_json=$(echo $audit_json_part1$audit_uid$audit_json_part2) 
}

get_pod_key(){
        pod_prefix=/registry/pod
        get_random_len_random_str
        pod_namespace=$random_len_str
        get_random_len_random_str
        pod_name=$random_len_str
        pod_key=$pod_prefix"/"$pod_namespace"/"$pod_name
}
get_audit_key(){
        audit_prefix=/registry/audit
        get_random_len_random_str
        audit_namespace=$random_len_str
        get_random_len_random_str
        audit_name=$random_len_str
        audit_key=$audit_prefix"/"$audit_namespace"/"$audit_name
}
                                                                                                                                   




@xiang90
Copy link
Contributor

xiang90 commented Jan 9, 2018

@WanLinghao

Your script tests etcd2 backend which supports 200MB at most, not etcd3. We have no plan to scale etcd2 backend, and k8s uses etcd3 by default in its recent versions. Closing this issue.

@xiang90 xiang90 closed this as completed Jan 9, 2018
@WanLinghao
Copy link
Author

@xiang90 how should I use etd3?
export ETCDCTL_API=3?
no need to change etcd server?

@xiang90
Copy link
Contributor

xiang90 commented Jan 9, 2018

yes

@WanLinghao
Copy link
Author

@xiang90 hello~
We have changed etcd V2 to v3.
However, we got trouble again.
When we put about 400,000 kvs, the server became unusable.
It gave "Error: etcdserver: mvcc: database space exceeded".At that time, the total size of etcd member dir was about 2.5GB.
After google, we adjust --quota-backend-bytes param to max value ,which is 8GB.And the server works again.
So here's my problem, since 400,000 kvs occupy 2.5GB disk space, can we suppose 8GB disk space to be about 1.2Mi kvs?
And you mentioned that you have a 5Mi pods test, the gap is obvious.

please give me some help or info.

@xiang90
Copy link
Contributor

xiang90 commented Jan 10, 2018

@WanLinghao Can you please do some simple calculation on your own? And try to figure out if the db size matches with what you put into it? We are not going to solve all your problem for you.

@spzala spzala added WIP and removed WIP labels Jan 12, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

No branches or pull requests

3 participants