Benchmarking is causing cluster to become unresponsive #7471

davissp14 · 2017-03-09T21:48:54Z

Running into a very strange issue when benchmarking my ETCD 3.1.0 3 node cluster.

./benchmark --endpoints=$ENDPOINTS --conns=100 --clients=1000 put --key-size=100 
--key-space-size=10 --sequential-keys --total=1000000 --val-size=20`

755885 / 1000000      Booooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooom                                                                              !  75.59% 1m34s

There has been a few times where this completes without issue, but most of the time it ends up putting my cluster into an unresponsive state, and the benchmarking progress comes to a halt. There's no logs that indicate an issue, it simply just becomes unresponsive until I finally start restarting nodes.

The one thing that seems to be consistent is the logs before becoming unresponsive.

pkg/fileutil: purged file /data/etcd/member/snap/0000000000000003-00000000000b2807.snap successfully
pkg/fileutil: purged file /data/etcd/member/snap/0000000000000003-00000000000b00f0.snap successfully
pkg/fileutil: purged file /data/etcd/member/snap/0000000000000003-00000000000b2807.snap successfully

Please let me know if I can provide you any additional information.

The text was updated successfully, but these errors were encountered:

gyuho · 2017-03-10T00:46:06Z

putting my cluster into an unresponsive state

Can you double-check with other client requests?
If benchmark command hangs, you can check etcd server logs or do some etcdctl commands

ETCDCTL_API=3 etcdctl endpoint health
ETCDCTL_API=3 etcdctl put foo bar
ETCDCTL_API=3 etcdctl get foo

We use this tool quite a lot, but I've never had this issue.

xiang90 · 2017-03-10T06:58:51Z

@davissp14 Can you provide us detailed steps to reproduce this issue (including your platform information)

davissp14 · 2017-03-10T15:15:48Z

I am running 3 nodes in a containerized environment (LXC) fronted with 2 etcd proxies, each node is on a separate ec2 host running Ubuntu 14.04. The 3 member nodes are running with 256MB memory (limit set by cgroups) and has no CPU / Disk / IOPS limits in place.

The only node that appears to lock up is the leader. Commands run just fine on all non-leader nodes, while commands on the leader hang indefinitely.

I am able to reproduce the issue consistency by simply running the benchmark command against an empty Etcd on the setup described above.
./benchmark --endpoints=$ENDPOINTS --conns=100 --clients=1000 put --key-size=100 --key-space-size=10 --sequential-keys --total=1000000 --val-size=20

**~ Boot Info ~**
etcdmain: etcd Version: 3.1.0
etcdmain: Git SHA: 8ba2897
etcdmain: Go Version: go1.7.4
etcdmain: Go OS/Arch: linux/amd64
etcdmain: setting maximum number of CPUs to 8, total number of available CPUs is 16
etcdmain: the server is already initialized as member before, starting as etcd member...
embed: listening for peers on http://10.46.172.132:2380
embed: listening for client requests on 10.46.172.132:2379
embed: listening for client requests on 127.0.0.1:2379
etcdserver: name = xxxx
etcdserver: data dir = /data/etcd
etcdserver: member dir = /data/etcd/member
etcdserver: heartbeat = 500ms
etcdserver: election = 5000ms
etcdserver: snapshot count = 10000

**~ Strace of hanging leader node~**

[pid    55] write(14, "36\r\n\0\0\0\0\0\0\0.\10\10\20\273\200\353\236\364\270\347\262\37\30\301\323\261\200\237\322\265"..., 60 <unfinished ...>
[pid    54] write(17, "36\r\n\0\0\0\0\0\0\0.\10\10\20\322\366\315\333\301\303\225\303\6\30\301\323\261\200\237\322\265"..., 60 <unfinished ...>
[pid    55] <... write resumed> )       = 60
[pid    54] <... write resumed> )       = 60
[pid    28] read(15, "34\r\n\0\0\0\0\0\0\0,\10\t\20\301\323\261\200\237\322\265\305]\30\273\200\353\236\364\270\347"..., 4096) = 58
[pid    28] read(15, 0xc421be8000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid    54] read(10, "34\r\n\0\0\0\0\0\0\0,\10\t\20\301\323\261\200\237\322\265\305]\30\322\366\315\333\301\303\225"..., 4096) = 58
[pid    54] read(10, 0xc420610000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid    29] write(16, "1\r\n\0\r\n", 6) = 6
[pid    54] write(17, "24\r\n\0\0\0\0\0\0\0\34\10\10\20\0\30\0 \0(\0000\0@\0J\10\22\6\n\0"..., 42) = 42
[pid    29] write(13, "1\r\n\0\r\n", 6) = 6
[pid    54] write(14, "24\r\n\0\0\0\0\0\0\0\34\10\10\20\0\30\0 \0(\0000\0@\0J\10\22\6\n\0"..., 42) = 42
[pid    29] write(20, "GET /version HTTP/1.1\r\nHost: 10."..., 106) = 106
[pid    28] read(20, "HTTP/1.1 200 OK\r\nContent-Type: a"..., 4096) = 152
[pid    28] read(20, 0xc421bfb000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid    28] write(21, "GET /version HTTP/1.1\r\nHost: 10."..., 106) = 106
[pid    55] read(21, "HTTP/1.1 200 OK\r\nContent-Type: a"..., 4096) = 152
[pid    55] read(21, 0xc421c03000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid    29] read(19, "1\r\n\0\r\n", 4096) = 6
[pid    29] read(19, 0xc4215c9000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid    29] read(15, "24\r\n\0\0\0\0\0\0\0\34\10\10\20\0\30\0 \0(\0000\0@\0J\10\22\6\n\0"..., 4096) = 42
[pid    29] read(15, 0xc421be8000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid    55] write(14, "36\r\n\0\0\0\0\0\0\0.\10\10\20\273\200\353\236\364\270\347\262\37\30\301\323\261\200\237\322\265"..., 60 <unfinished ...>
[pid    28] write(17, "36\r\n\0\0\0\0\0\0\0.\10\10\20\322\366\315\333\301\303\225\303\6\30\301\323\261\200\237\322\265"..., 60 <unfinished ...>
[pid    55] <... write resumed> )       = 60
[pid    28] <... write resumed> )       = 60
[pid    29] read(15, "34\r\n\0\0\0\0\0\0\0,\10\t\20\301\323\261\200\237\322\265\305]\30\273\200\353\236\364\270\347"..., 4096) = 58
[pid    28] read(10, "34\r\n\0\0\0\0\0\0\0,\10\t\20\301\323\261\200\237\322\265\305]\30\322\366\315\333\301\303\225"..., 4096) = 58
[pid    29] read(15, 0xc421be8000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid    28] read(10, 0xc420610000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid    29] read(12, "1\r\n\0\r\n", 4096) = 6
[pid    29] read(12, 0xc420591000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid    29] read(49, "GET /raft/probing HTTP/1.1\r\nHost"..., 4096) = 111
[pid    29] write(49, "HTTP/1.1 200 OK\r\nDate: Fri, 10 M"..., 168) = 168
[pid    29] read(49, 0xc420db3000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid    29] read(10, "24\r\n\0\0\0\0\0\0\0\34\10\10\20\0\30\0 \0(\0000\0@\0J\10\22\6\n\0"..., 4096) = 42
[pid    29] read(10, 0xc420610000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid    55] write(17, "36\r\n\0\0\0\0\0\0\0.\10\10\20\322\366\315\333\301\303\225\303\6\30\301\323\261\200\237\322\265"..., 60 <unfinished ...>
[pid    54] write(14, "36\r\n\0\0\0\0\0\0\0.\10\10\20\273\200\353\236\364\270\347\262\37\30\301\323\261\200\237\322\265"..., 60 <unfinished ...>
[pid    55] <... write resumed> )       = 60
[pid    54] <... write resumed> )       = 60
[pid    29] read(15, "34\r\n\0\0\0\0\0\0\0,\10\t\20\301\323\261\200\237\322\265\305]\30\273\200\353\236\364\270\347"..., 4096) = 58
[pid    54] read(10, "34\r\n\0\0\0\0\0\0\0,\10\t\20\301\323\261\200\237\322\265\305]\30\322\366\315\333\301\303\225"..., 4096) = 58
[pid    29] read(15, 0xc421be8000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid    54] read(10, 0xc420610000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid    55] write(18, "GET /raft/probing HTTP/1.1\r\nHost"..., 111) = 111
[pid    54] read(18, "HTTP/1.1 200 OK\r\nDate: Fri, 10 M"..., 4096) = 168
[pid    54] read(18, 0xc420618000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid    55] write(48, "GET /raft/probing HTTP/1.1\r\nHost"..., 111) = 111
[pid    29] read(48, "HTTP/1.1 200 OK\r\nDate: Fri, 10 M"..., 4096) = 168
[pid    29] read(48, 0xc421c0b000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid    55] write(17, "36\r\n\0\0\0\0\0\0\0.\10\10\20\322\366\315\333\301\303\225\303\6\30\301\323\261\200\237\322\265"..., 60) = 60
[pid    28] write(14, "36\r\n\0\0\0\0\0\0\0.\10\10\20\273\200\353\236\364\270\347\262\37\30\301\323\261\200\237\322\265"..., 60) = 60
[pid    54] read(15, "34\r\n\0\0\0\0\0\0\0,\10\t\20\301\323\261\200\237\322\265\305]\30\273\200\353\236\364\270\347"..., 4096) = 58
[pid    28] read(10, "34\r\n\0\0\0\0\0\0\0,\10\t\20\301\323\261\200\237\322\265\305]\30\322\366\315\333\301\303\225"..., 4096) = 58
[pid    54] read(15, 0xc421be8000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid    28] read(10, 0xc420610000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid    54] read(50, "GET /raft/probing HTTP/1.1\r\nHost"..., 4096) = 111
[pid    54] write(50, "HTTP/1.1 200 OK\r\nDate: Fri, 10 M"..., 168) = 168
[pid    54] read(50, 0xc421c13000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid    29] write(17, "24\r\n\0\0\0\0\0\0\0\34\10\10\20\0\30\0 \0(\0000\0@\0J\10\22\6\n\0"..., 42) = 42
[pid    55] write(16, "1\r\n\0\r\n", 6) = 6
[pid    29] write(13, "1\r\n\0\r\n", 6) = 6
[pid    55] write(14, "24\r\n\0\0\0\0\0\0\0\34\10\10\20\0\30\0 \0(\0000\0@\0J\10\22\6\n\0"..., 42) = 42
[pid    54] read(19, "1\r\n\0\r\n", 4096) = 6
[pid    54] read(19, 0xc4215c9000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid    54] read(15, "24\r\n\0\0\0\0\0\0\0\34\10\10\20\0\30\0 \0(\0000\0@\0J\10\22\6\n\0"..., 4096) = 42
[pid    54] read(15, 0xc421be8000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid    29] write(17, "36\r\n\0\0\0\0\0\0\0.\10\10\20\322\366\315\333\301\303\225\303\6\30\301\323\261\200\237\322\265"..., 60 <unfinished ...>
[pid    28] write(14, "36\r\n\0\0\0\0\0\0\0.\10\10\20\273\200\353\236\364\270\347\262\37\30\301\323\261\200\237\322\265"..., 60 <unfinished ...>
[pid    29] <... write resumed> )       = 60
[pid    28] <... write resumed> )       = 60
[pid    54] read(15, "34\r\n\0\0\0\0\0\0\0,\10\t\20\301\323\261\200\237\322\265\305]\30\273\200\353\236\364\270\347"..., 4096) = 58
[pid    54] read(15, 0xc421be8000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid    54] read(10, "34\r\n\0\0\0\0\0\0\0,\10\t\20\301\323\261\200\237\322\265\305]\30\322\366\315\333\301\303\225"..., 4096) = 58
[pid    54] read(10, 0xc420610000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid    28] read(12, "1\r\n\0\r\n", 4096) = 6
[pid    28] read(12, 0xc420591000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid    28] read(10, "24\r\n\0\0\0\0\0\0\0\34\10\10\20\0\30\0 \0(\0000\0@\0J\10\22\6\n\0"..., 4096) = 42
[pid    28] read(10, 0xc420610000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid    29] write(14, "36\r\n\0\0\0\0\0\0\0.\10\10\20\273\200\353\236\364\270\347\262\37\30\301\323\261\200\237\322\265"..., 60) = 60
[pid    54] write(17, "36\r\n\0\0\0\0\0\0\0.\10\10\20\322\366\315\333\301\303\225\303\6\30\301\323\261\200\237\322\265"..., 60) = 60
[pid    28] read(15, "34\r\n\0\0\0\0\0\0\0,\10\t\20\301\323\261\200\237\322\265\305]\30\273\200\353\236\364\270\347"..., 4096) = 58
[pid    28] read(15,  <unfinished ...>
[pid    54] read(10,  <unfinished ...>
[pid    28] <... read resumed> 0xc421be8000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid    54] <... read resumed> "34\r\n\0\0\0\0\0\0\0,\10\t\20\301\323\261\200\237\322\265\305]\30\322\366\315\333\301\303\225"..., 4096) = 58
[pid    54] read(10, 0xc420610000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid    29] write(14, "36\r\n\0\0\0\0\0\0\0.\10\10\20\273\200\353\236\364\270\347\262\37\30\301\323\261\200\237\322\265"..., 60) = 60
[pid    55] write(17, "36\r\n\0\0\0\0\0\0\0.\10\10\20\322\366\315\333\301\303\225\303\6\30\301\323\261\200\237\322\265"..., 60) = 60
[pid    28] read(15, "34\r\n\0\0\0\0\0\0\0,\10\t\20\301\323\261\200\237\322\265\305]\30\273\200\353\236\364\270\347"..., 4096) = 58
[pid    28] read(15, 0xc421be8000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid    55] read(10, "34\r\n\0\0\0\0\0\0\0,\10\t\20\301\323\261\200\237\322\265\305]\30\322\366\315\333\301\303\225"..., 4096) = 58
[pid    55] read(10, 0xc420610000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid    55] write(16, "1\r\n\0\r\n", 6) = 6
[pid    54] write(17, "24\r\n\0\0\0\0\0\0\0\34\10\10\20\0\30\0 \0(\0000\0@\0J\10\22\6\n\0"..., 42) = 42
[pid    55] write(13, "1\r\n\0\r\n", 6) = 6
[pid    54] write(14, "24\r\n\0\0\0\0\0\0\0\34\10\10\20\0\30\0 \0(\0000\0@\0J\10\22\6\n\0"..., 42) = 42
[pid    55] write(14, "36\r\n\0\0\0\0\0\0\0.\10\10\20\273\200\353\236\364\270\347\262\37\30\301\323\261\200\237\322\265"..., 60) = 60
[pid    29] write(17, "36\r\n\0\0\0\0\0\0\0.\10\10\20\322\366\315\333\301\303\225\303\6\30\301\323\261\200\237\322\265"..., 60) = 60
[pid    28] read(15, "34\r\n\0\0\0\0\0\0\0,\10\t\20\301\323\261\200\237\322\265\305]\30\273\200\353\236\364\270\347"..., 4096) = 58
[pid    28] read(15, 0xc421be8000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid    54] read(10, "34\r\n\0\0\0\0\0\0\0,\10\t\20\301\323\261\200\237\322\265\305]\30\322\366\315\333\301\303\225"..., 4096) = 58
[pid    54] read(10, 0xc420610000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid    28] read(19, "1\r\n\0\r\n", 4096) = 6
[pid    28] read(19, 0xc4215c9000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid    28] read(15, "24\r\n\0\0\0\0\0\0\0\34\10\10\20\0\30\0 \0(\0000\0@\0J\10\22\6\n\0"..., 4096) = 42
[pid    28] read(15, 0xc421be8000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid    29] write(17, "36\r\n\0\0\0\0\0\0\0.\10\10\20\322\366\315\333\301\303\225\303\6\30\301\323\261\200\237\322\265"..., 60 <unfinished ...>
[pid    55] write(14, "36\r\n\0\0\0\0\0\0\0.\10\10\20\273\200\353\236\364\270\347\262\37\30\301\323\261\200\237\322\265"..., 60 <unfinished ...>
[pid    29] <... write resumed> )       = 60
[pid    55] <... write resumed> )       = 60
[pid    28] read(15, "34\r\n\0\0\0\0\0\0\0,\10\t\20\301\323\261\200\237\322\265\305]\30\273\200\353\236\364\270\347"..., 4096) = 58
[pid    55] read(10, "34\r\n\0\0\0\0\0\0\0,\10\t\20\301\323\261\200\237\322\265\305]\30\322\366\315\333\301\303\225"..., 4096) = 58
[pid    28] read(15, 0xc421be8000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid    55] read(10, 0xc420610000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid    55] read(12, "1\r\n\0\r\n", 4096) = 6
[pid    55] read(12, 0xc420591000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid    55] read(10, "24\r\n\0\0\0\0\0\0\0\34\10\10\20\0\30\0 \0(\0000\0@\0J\10\22\6\n\0"..., 4096) = 42
[pid    55] read(10, 0xc420610000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid    54] write(20, "GET /version HTTP/1.1\r\nHost: 10."..., 106) = 106
[pid    55] read(20, "HTTP/1.1 200 OK\r\nContent-Type: a"..., 4096) = 152
[pid    55] read(20, 0xc421bfb000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid    55] write(21, "GET /version HTTP/1.1\r\nHost: 10."..., 106) = 106
[pid    29] read(21, "HTTP/1.1 200 OK\r\nContent-Type: a"..., 4096) = 152
[pid    29] read(21, 0xc421c03000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid    29] write(17, "36\r\n\0\0\0\0\0\0\0.\10\10\20\322\366\315\333\301\303\225\303\6\30\301\323\261\200\237\322\265"..., 60 <unfinished ...>
[pid    54] write(14, "36\r\n\0\0\0\0\0\0\0.\10\10\20\273\200\353\236\364\270\347\262\37\30\301\323\261\200\237\322\265"..., 60) = 60
[pid    29] <... write resumed> )       = 60
[pid    55] read(15, "34\r\n\0\0\0\0\0\0\0,\10\t\20\301\323\261\200\237\322\265\305]\30\273\200\353\236\364\270\347"..., 4096) = 58
[pid    55] read(15, 0xc421be8000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid    55] read(10, "34\r\n\0\0\0\0\0\0\0,\10\t\20\301\323\261\200\237\322\265\305]\30\322\366\315\333\301\303\225"..., 4096) = 58
[pid    55] read(10, 0xc420610000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
[pid    29] read(49, "GET /raft/probing HTTP/1.1\r\nHost"..., 4096) = 111
[pid    29] write(49, "HTTP/1.1 200 OK\r\nDate: Fri, 10 M"..., 168) = 168
[pid    29] read(49, 0xc420db3000, 4096) = -1 EAGAIN (Resource temporarily unavailable)

davissp14 · 2017-03-10T18:58:18Z

I went ahead and updated to 3.1.2 to see if it helped. While it's not hanging like it was in 3.1.0, I am now seeing a panic.

Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19  | panic: runtime error: invalid memory address or nil pointer dereference
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19  | [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x8bdda8]
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19  |
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19  | goroutine 2184 [running]:
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19  | panic(0xd15980, 0xc42000c0e0)
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19  | 	/usr/local/go/src/runtime/panic.go:500 +0x1a1
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19  | github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth.(*authStore).IsAdminPermitted(0xc4201f64d0, 0x0, 0x0, 0x0)
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19  | 	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth/store.go:772 +0xc8
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19  | github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.(*authMaintenanceServer).isAuthenticated(0xc420297c80, 0x7f88486169d0, 0xc4228e27e0, 0x12e0ec0, 0xc4215369c0)
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19  | 	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/maintenance.go:160 +0xae
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19  | github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.(*authMaintenanceServer).Status(0xc420297c80, 0x7f88486169d0, 0xc4228e27e0, 0x132c548, 0x5, 0xe588f1, 0x18)
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19  | 	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/maintenance.go:188 +0x43
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19  | github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb._Maintenance_Status_Handler.func1(0x7f88486169d0, 0xc4228e27e0, 0xd9f580, 0x132c548, 0xc420a90730, 0xc421536a38, 0x9bc1a6, 0xc4228e27e0)
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19  | 	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.go:3025 +0xa2
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19  | github.com/coreos/etcd/cmd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.UnaryServerInterceptor(0x7f88486169d0, 0xc4228e27e0, 0xd9f580, 0x132c548, 0xc421a4b4c0, 0xc421a4b4e0, 0x40f9e8, 0x20, 0xd479a0, 0x1)
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19  | 	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server.go:29 +0xa4
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19  | github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.newUnaryInterceptor.func1(0x7f88486169d0, 0xc4228e27e0, 0xd9f580, 0x132c548, 0xc421a4b4c0, 0xc421a4b4e0, 0x0, 0xc420029000, 0x4f8934, 0xc421536b28)
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19  | 	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/interceptor.go:57 +0xc3
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19  | github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb._Maintenance_Status_Handler(0xd6a840, 0xc420297c80, 0x7f88486169d0, 0xc4228e27e0, 0xc420a90690, 0xc4202afe60, 0x0, 0x0, 0xc42008b6b0, 0x7f8ac87bdc08)
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19  | 	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.go:3027 +0x166
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19  | github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc42043e180, 0x12e3500, 0xc4201930e0, 0xc4215dc5a0, 0xc4202b5200, 0x1301398, 0x0, 0x0, 0x0)
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19  | 	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:638 +0xc14
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19  | github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).handleStream(0xc42043e180, 0x12e3500, 0xc4201930e0, 0xc4215dc5a0, 0x0)
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19  | 	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:796 +0x6b0
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19  | github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc421a2ec80, 0xc42043e180, 0x12e3500, 0xc4201930e0, 0xc4215dc5a0)
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19  | 	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:449 +0xab
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19  | created by github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: etcd.19  | 	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:450 +0xa3
Mar 10 18:54:52 sl-us-dal-9-memory etcd746: start.13 | etcd.19 process exited.

gyuho · 2017-03-10T19:02:42Z

Commands run just fine on all non-leader nodes, while commands on the leader hang indefinitely.

Which commands? Do you mean benchmark commands or etcdctl commands? Trying to figure out whether the etcd server is hanging or benchmark commands.

heyitsanthony · 2017-03-10T19:08:11Z

@davissp14 Is this cluster totally fresh? That path would only crash if auth is enabled...

Is the hang sensitive to the arguments? Does --conns=10 work? --clients=100? I ran it locally and the memory utilization is under 150MB.

Commands run just fine on all non-leader nodes, while commands on the leader hang indefinitely.

Commands sent to the leader node or commands running on the leader node? ETCDCTL_API=3 etcdctl get foo or ...? What is the output from ETCDCTL_API=3 ./bin/etcdctl --endpoints=$ENDPOINTS endpoint health?

Thanks!

davissp14 · 2017-03-10T19:09:08Z

@gyuho Both the benchmark commands as well as the etcdctl commands hang on the leader node in 3.1.0. I am able to write to the non-leader nodes without issues, which seems strange. It appears that the server-to-server communication running on 2380 is working just fine, it's just client requests on 2379 are being blocked.

davissp14 · 2017-03-10T19:41:38Z

The last benchmark I ran actually managed to throw 2 of the 3 nodes into an unresponsive state (leader included). I am still able to read and write to the responsive node without problems.

root $ETCDCTL_API=3 etcdctl --endpoints=10.11.221.131:2379,10.11.221.130:2379,10.11.221.132:2379 --user=root:XXX endpoint health
10.11.221.132:2379 is healthy: successfully committed proposal: took = 3.081202ms

After reporting that the 1 responsive node is healthy, the endpoint health command just hangs. Running this on all 3 member nodes as well as the 2 proxies and each produce the same result.

@heyitsanthony This was a fresh cluster with auth enabled. I will play with the --conns value and see if it makes any difference.

UPDATE
@heyitsanthony Changing --conns 10 did not make a difference :(

davissp14 · 2017-03-13T21:50:55Z

Here is the output of curl localhost:2379/debug/pprof/goroutine?debug=1 from one of the unresponsive nodes.

goroutine profile: total 1163
996 @ 0x42cc8a 0x42cd7e 0x43ce8d 0x43cb20 0x74b39d 0x8bcace 0x8bfd99 0x5f6d9c 0x5f749a 0x5f2508 0x796788 0x79a3f2 0x70f6a2 0x61d224 0x79d783 0x6b2ae6 0x643254 0x644ef0 0x64a9fb 0x45de01
#	0x43cb1f	sync.runtime_Semacquire+0x2f												/usr/local/go/src/runtime/sema.go:47
#	0x74b39c	sync.(*RWMutex).RLock+0x5c												/usr/local/go/src/sync/rwmutex.go:43
#	0x8bcacd	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth.(*authStore).AuthInfoFromToken+0x4d			/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth/store.go:649
#	0x8bfd98	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth.(*authStore).AuthInfoFromCtx+0x108			/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth/store.go:989
#	0x5f6d9b	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).processInternalRaftRequestOnce+0x13b	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/v3_server.go:655
#	0x5f7499	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).processInternalRaftRequest+0x89	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/v3_server.go:703
#	0x5f2507	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).Put+0xc7				/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/v3_server.go:142
#	0x796787	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.(*kvServer).Put+0x87			/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/key.go:67
#	0x79a3f1	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.(*quotaKVServer).Put+0xa1			/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/quota.go:62
#	0x70f6a1	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb._KV_Put_Handler.func1+0xa1		/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.go:2305
#	0x61d223	github.com/coreos/etcd/cmd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.UnaryServerInterceptor+0xa3		/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server.go:29
#	0x79d782	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.newUnaryInterceptor.func1+0xc2		/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/interceptor.go:57
#	0x6b2ae5	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb._KV_Put_Handler+0x165			/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.go:2307
#	0x643253	github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).processUnaryRPC+0xc13				/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:638
#	0x644eef	github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).handleStream+0x6af					/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:796
#	0x64a9fa	github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1+0xaa				/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:449

14 @ 0x42cc8a 0x42754b 0x426ca9 0x4f88b8 0x4f8924 0x4fa161 0x50b1e0 0x60e1eb 0x736d93 0x866aec 0x86714c 0x744074 0x7441d8 0x9283cb 0x928c7e 0x9b662f 0x9af450 0x641b10 0x64196a 0x641550 0x45de01
#	0x426ca8	net.runtime_pollWait+0x58										/usr/local/go/src/runtime/netpoll.go:160
#	0x4f88b7	net.(*pollDesc).wait+0x37										/usr/local/go/src/net/fd_poll_runtime.go:73
#	0x4f8923	net.(*pollDesc).waitRead+0x33										/usr/local/go/src/net/fd_poll_runtime.go:78
#	0x4fa160	net.(*netFD).Read+0x1a0											/usr/local/go/src/net/fd_unix.go:243
#	0x50b1df	net.(*conn).Read+0x6f											/usr/local/go/src/net/net.go:173
#	0x60e1ea	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/transport.(*limitListenerConn).Read+0x6a	<autogenerated>:16
#	0x736d92	github.com/coreos/etcd/cmd/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read+0x112			/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/cockroachdb/cmux/cmux.go:204
#	0x866aeb	bufio.(*Reader).fill+0x10b										/usr/local/go/src/bufio/bufio.go:97
#	0x86714b	bufio.(*Reader).Read+0x1bb										/usr/local/go/src/bufio/bufio.go:209
#	0x744073	io.ReadAtLeast+0xa3											/usr/local/go/src/io/io.go:307
#	0x7441d7	io.ReadFull+0x57											/usr/local/go/src/io/io.go:325
#	0x9283ca	github.com/coreos/etcd/cmd/vendor/golang.org/x/net/http2.readFrameHeader+0x7a				/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/golang.org/x/net/http2/frame.go:237
#	0x928c7d	github.com/coreos/etcd/cmd/vendor/golang.org/x/net/http2.(*Framer).ReadFrame+0x8d			/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/golang.org/x/net/http2/frame.go:469
#	0x9b662e	github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport.(*framer).readFrame+0x2e		/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport/http_util.go:508
#	0x9af44f	github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport.(*http2Server).HandleStreams+0x1ff	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport/http2_server.go:260
#	0x641b0f	github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).serveStreams+0x15f			/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:451
#	0x641969	github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).serveNewHTTP2Transport+0x3d9		/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:438
#	0x64154f	github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).handleRawConn+0x46f			/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:415

14 @ 0x42cc8a 0x43c2b5 0x43b04c 0x9b365c 0x45de01
#	0x9b365b	github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport.(*http2Server).controller+0x69b	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport/http2_server.go:674

12 @ 0x42cc8a 0x42754b 0x426ca9 0x4f88b8 0x4f8924 0x4fa161 0x50b1e0 0x866aec 0x86714c 0x744074 0x7441d8 0x9283cb 0x928c7e 0x9b662f 0x9ac7c6 0x45de01
#	0x426ca8	net.runtime_pollWait+0x58									/usr/local/go/src/runtime/netpoll.go:160
#	0x4f88b7	net.(*pollDesc).wait+0x37									/usr/local/go/src/net/fd_poll_runtime.go:73
#	0x4f8923	net.(*pollDesc).waitRead+0x33									/usr/local/go/src/net/fd_poll_runtime.go:78
#	0x4fa160	net.(*netFD).Read+0x1a0										/usr/local/go/src/net/fd_unix.go:243
#	0x50b1df	net.(*conn).Read+0x6f										/usr/local/go/src/net/net.go:173
#	0x866aeb	bufio.(*Reader).fill+0x10b									/usr/local/go/src/bufio/bufio.go:97
#	0x86714b	bufio.(*Reader).Read+0x1bb									/usr/local/go/src/bufio/bufio.go:209
#	0x744073	io.ReadAtLeast+0xa3										/usr/local/go/src/io/io.go:307
#	0x7441d7	io.ReadFull+0x57										/usr/local/go/src/io/io.go:325
#	0x9283ca	github.com/coreos/etcd/cmd/vendor/golang.org/x/net/http2.readFrameHeader+0x7a			/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/golang.org/x/net/http2/frame.go:237
#	0x928c7d	github.com/coreos/etcd/cmd/vendor/golang.org/x/net/http2.(*Framer).ReadFrame+0x8d		/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/golang.org/x/net/http2/frame.go:469
#	0x9b662e	github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport.(*framer).readFrame+0x2e	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport/http_util.go:508
#	0x9ac7c5	github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport.(*http2Client).reader+0xb5	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport/http2_client.go:937

12 @ 0x42cc8a 0x43c2b5 0x43b04c 0x63c6ae 0x64a4de 0x45de01
#	0x63c6ad	github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*addrConn).transportMonitor+0x63d		/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/clientconn.go:751
#	0x64a4dd	github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*ClientConn).resetAddrConn.func1+0x1dd	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/clientconn.go:506

12 @ 0x42cc8a 0x43c2b5 0x43b04c 0x9ad76d 0x45de01
#	0x9ad76c	github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport.(*http2Client).controller+0x59c	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport/http2_client.go:1015

8 @ 0x42cc8a 0x43c2b5 0x43b04c 0x57c37d 0x45de01
#	0x57c37c	net/http.(*persistConn).writeLoop+0x3bc	/usr/local/go/src/net/http/transport.go:1646

8 @ 0x42cc8a 0x43c2b5 0x43b04c 0x7821b7 0x45de01
#	0x7821b6	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp.(*pipeline).handle+0x686	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp/pipeline.go:80

4 @ 0x42cc8a 0x42754b 0x426ca9 0x4f88b8 0x4f8924 0x4fa161 0x50b1e0 0x60bb8d 0x60f43b 0x579bb4 0x866aec 0x866cd2 0x57a7d1 0x45de01
#	0x426ca8	net.runtime_pollWait+0x58									/usr/local/go/src/runtime/netpoll.go:160
#	0x4f88b7	net.(*pollDesc).wait+0x37									/usr/local/go/src/net/fd_poll_runtime.go:73
#	0x4f8923	net.(*pollDesc).waitRead+0x33									/usr/local/go/src/net/fd_poll_runtime.go:78
#	0x4fa160	net.(*netFD).Read+0x1a0										/usr/local/go/src/net/fd_unix.go:243
#	0x50b1df	net.(*conn).Read+0x6f										/usr/local/go/src/net/net.go:173
#	0x60bb8c	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/transport.timeoutConn.Read+0x5c	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/transport/timeout_conn.go:43
#	0x60f43a	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/transport.(*timeoutConn).Read+0x8a	<autogenerated>:42
#	0x579bb3	net/http.(*persistConn).Read+0x153								/usr/local/go/src/net/http/transport.go:1261
#	0x866aeb	bufio.(*Reader).fill+0x10b									/usr/local/go/src/bufio/bufio.go:97
#	0x866cd1	bufio.(*Reader).Peek+0x61									/usr/local/go/src/bufio/bufio.go:129
#	0x57a7d0	net/http.(*persistConn).readLoop+0x1a0								/usr/local/go/src/net/http/transport.go:1418

4 @ 0x42cc8a 0x42cd7e 0x403838 0x4035fd 0x8bcb99 0x8bfd99 0x5f6d9c 0x5f749a 0x5f2508 0x796788 0x79a3f2 0x70f6a2 0x61d224 0x79d783 0x6b2ae6 0x643254 0x644ef0 0x64a9fb 0x45de01
#	0x8bcb98	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth.(*authStore).AuthInfoFromToken+0x118			/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth/store.go:653
#	0x8bfd98	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth.(*authStore).AuthInfoFromCtx+0x108			/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth/store.go:989
#	0x5f6d9b	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).processInternalRaftRequestOnce+0x13b	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/v3_server.go:655
#	0x5f7499	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).processInternalRaftRequest+0x89	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/v3_server.go:703
#	0x5f2507	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).Put+0xc7				/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/v3_server.go:142
#	0x796787	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.(*kvServer).Put+0x87			/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/key.go:67
#	0x79a3f1	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.(*quotaKVServer).Put+0xa1			/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/quota.go:62
#	0x70f6a1	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb._KV_Put_Handler.func1+0xa1		/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.go:2305
#	0x61d223	github.com/coreos/etcd/cmd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.UnaryServerInterceptor+0xa3		/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server.go:29
#	0x79d782	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.newUnaryInterceptor.func1+0xc2		/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/interceptor.go:57
#	0x6b2ae5	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb._KV_Put_Handler+0x165			/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.go:2307
#	0x643253	github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).processUnaryRPC+0xc13				/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:638
#	0x644eef	github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).handleStream+0x6af					/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:796
#	0x64a9fa	github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1+0xaa				/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:449

4 @ 0x42cc8a 0x42cd7e 0x40452f 0x4041f5 0x77ccb5 0x56ab3f 0x56ab3f 0x56b62d 0x567fd7 0x45de01
#	0x77ccb4	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp.(*streamHandler).ServeHTTP+0xba4	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp/http.go:323
#	0x56ab3e	net/http.(*ServeMux).ServeHTTP+0x7e									/usr/local/go/src/net/http/server.go:2022
#	0x56ab3e	net/http.(*ServeMux).ServeHTTP+0x7e									/usr/local/go/src/net/http/server.go:2022
#	0x56b62c	net/http.serverHandler.ServeHTTP+0x7c									/usr/local/go/src/net/http/server.go:2202
#	0x567fd6	net/http.(*conn).serve+0x4b6										/usr/local/go/src/net/http/server.go:1579

4 @ 0x42cc8a 0x43c2b5 0x43b04c 0x57aff9 0x45de01
#	0x57aff8	net/http.(*persistConn).readLoop+0x9c8	/usr/local/go/src/net/http/transport.go:1541

4 @ 0x42cc8a 0x43c2b5 0x43b04c 0x786519 0x45de01
#	0x786518	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp.(*streamWriter).run+0x17e8	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp/stream.go:149

2 @ 0x42cc8a 0x42754b 0x426ca9 0x4f88b8 0x4f8924 0x4fa161 0x50b1e0 0x60bb8d 0x60f43b 0x562b44 0x866aec 0x867795 0x867917 0x86347e 0x86327f 0x55d5b5 0x563ebd 0x567ef3 0x45de01
#	0x426ca8	net.runtime_pollWait+0x58									/usr/local/go/src/runtime/netpoll.go:160
#	0x4f88b7	net.(*pollDesc).wait+0x37									/usr/local/go/src/net/fd_poll_runtime.go:73
#	0x4f8923	net.(*pollDesc).waitRead+0x33									/usr/local/go/src/net/fd_poll_runtime.go:78
#	0x4fa160	net.(*netFD).Read+0x1a0										/usr/local/go/src/net/fd_unix.go:243
#	0x50b1df	net.(*conn).Read+0x6f										/usr/local/go/src/net/net.go:173
#	0x60bb8c	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/transport.timeoutConn.Read+0x5c	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/transport/timeout_conn.go:43
#	0x60f43a	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/transport.(*timeoutConn).Read+0x8a	<autogenerated>:42
#	0x562b43	net/http.(*connReader).Read+0x143								/usr/local/go/src/net/http/server.go:586
#	0x866aeb	bufio.(*Reader).fill+0x10b									/usr/local/go/src/bufio/bufio.go:97
#	0x867794	bufio.(*Reader).ReadSlice+0xb4									/usr/local/go/src/bufio/bufio.go:330
#	0x867916	bufio.(*Reader).ReadLine+0x36									/usr/local/go/src/bufio/bufio.go:359
#	0x86347d	net/textproto.(*Reader).readLineSlice+0x5d							/usr/local/go/src/net/textproto/reader.go:55
#	0x86327e	net/textproto.(*Reader).ReadLine+0x2e								/usr/local/go/src/net/textproto/reader.go:36
#	0x55d5b4	net/http.readRequest+0xa4									/usr/local/go/src/net/http/request.go:793
#	0x563ebc	net/http.(*conn).readRequest+0x10c								/usr/local/go/src/net/http/server.go:765
#	0x567ef2	net/http.(*conn).serve+0x3d2									/usr/local/go/src/net/http/server.go:1532

2 @ 0x42cc8a 0x42754b 0x426ca9 0x4f88b8 0x4f8924 0x4fa161 0x50b1e0 0x60bb8d 0x60f43b 0x579bb4 0x866aec 0x867795 0x87db64 0x87d5a2 0x87d80b 0x571771 0x5716b6 0x57d816 0x744074 0x7441d8 0x77e60a 0x78764e 0x787154 0x45de01
#	0x426ca8	net.runtime_pollWait+0x58										/usr/local/go/src/runtime/netpoll.go:160
#	0x4f88b7	net.(*pollDesc).wait+0x37										/usr/local/go/src/net/fd_poll_runtime.go:73
#	0x4f8923	net.(*pollDesc).waitRead+0x33										/usr/local/go/src/net/fd_poll_runtime.go:78
#	0x4fa160	net.(*netFD).Read+0x1a0											/usr/local/go/src/net/fd_unix.go:243
#	0x50b1df	net.(*conn).Read+0x6f											/usr/local/go/src/net/net.go:173
#	0x60bb8c	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/transport.timeoutConn.Read+0x5c		/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/transport/timeout_conn.go:43
#	0x60f43a	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/transport.(*timeoutConn).Read+0x8a		<autogenerated>:42
#	0x579bb3	net/http.(*persistConn).Read+0x153									/usr/local/go/src/net/http/transport.go:1261
#	0x866aeb	bufio.(*Reader).fill+0x10b										/usr/local/go/src/bufio/bufio.go:97
#	0x867794	bufio.(*Reader).ReadSlice+0xb4										/usr/local/go/src/bufio/bufio.go:330
#	0x87db63	net/http/internal.readChunkLine+0x33									/usr/local/go/src/net/http/internal/chunked.go:110
#	0x87d5a1	net/http/internal.(*chunkedReader).beginChunk+0x31							/usr/local/go/src/net/http/internal/chunked.go:47
#	0x87d80a	net/http/internal.(*chunkedReader).Read+0x7a								/usr/local/go/src/net/http/internal/chunked.go:77
#	0x571770	net/http.(*body).readLocked+0x60									/usr/local/go/src/net/http/transfer.go:651
#	0x5716b5	net/http.(*body).Read+0xf5										/usr/local/go/src/net/http/transfer.go:643
#	0x57d815	net/http.(*bodyEOFSignal).Read+0x115									/usr/local/go/src/net/http/transport.go:1974
#	0x744073	io.ReadAtLeast+0xa3											/usr/local/go/src/io/io.go:307
#	0x7441d7	io.ReadFull+0x57											/usr/local/go/src/io/io.go:325
#	0x77e609	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp.(*msgAppV2Decoder).decode+0xd9	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp/msgappv2_codec.go:179
#	0x78764d	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp.(*streamReader).decodeLoop+0x20d	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp/stream.go:358
#	0x787153	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp.(*streamReader).run+0x6f3		/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp/stream.go:311

2 @ 0x42cc8a 0x42754b 0x426ca9 0x4f88b8 0x4f8924 0x4fa161 0x50b1e0 0x60bb8d 0x60f43b 0x579bb4 0x866aec 0x867795 0x87db64 0x87d5a2 0x87d80b 0x571771 0x5716b6 0x57d816 0x744074 0x7441d8 0x86e650 0x77d912 0x78764e 0x787154 0x45de01
#	0x426ca8	net.runtime_pollWait+0x58										/usr/local/go/src/runtime/netpoll.go:160
#	0x4f88b7	net.(*pollDesc).wait+0x37										/usr/local/go/src/net/fd_poll_runtime.go:73
#	0x4f8923	net.(*pollDesc).waitRead+0x33										/usr/local/go/src/net/fd_poll_runtime.go:78
#	0x4fa160	net.(*netFD).Read+0x1a0											/usr/local/go/src/net/fd_unix.go:243
#	0x50b1df	net.(*conn).Read+0x6f											/usr/local/go/src/net/net.go:173
#	0x60bb8c	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/transport.timeoutConn.Read+0x5c		/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/transport/timeout_conn.go:43
#	0x60f43a	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/transport.(*timeoutConn).Read+0x8a		<autogenerated>:42
#	0x579bb3	net/http.(*persistConn).Read+0x153									/usr/local/go/src/net/http/transport.go:1261
#	0x866aeb	bufio.(*Reader).fill+0x10b										/usr/local/go/src/bufio/bufio.go:97
#	0x867794	bufio.(*Reader).ReadSlice+0xb4										/usr/local/go/src/bufio/bufio.go:330
#	0x87db63	net/http/internal.readChunkLine+0x33									/usr/local/go/src/net/http/internal/chunked.go:110
#	0x87d5a1	net/http/internal.(*chunkedReader).beginChunk+0x31							/usr/local/go/src/net/http/internal/chunked.go:47
#	0x87d80a	net/http/internal.(*chunkedReader).Read+0x7a								/usr/local/go/src/net/http/internal/chunked.go:77
#	0x571770	net/http.(*body).readLocked+0x60									/usr/local/go/src/net/http/transfer.go:651
#	0x5716b5	net/http.(*body).Read+0xf5										/usr/local/go/src/net/http/transfer.go:643
#	0x57d815	net/http.(*bodyEOFSignal).Read+0x115									/usr/local/go/src/net/http/transport.go:1974
#	0x744073	io.ReadAtLeast+0xa3											/usr/local/go/src/io/io.go:307
#	0x7441d7	io.ReadFull+0x57											/usr/local/go/src/io/io.go:325
#	0x86e64f	encoding/binary.Read+0x11f										/usr/local/go/src/encoding/binary/binary.go:168
#	0x77d911	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp.(*messageDecoder).decode+0x111	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp/msg_codec.go:53
#	0x78764d	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp.(*streamReader).decodeLoop+0x20d	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp/stream.go:358
#	0x787153	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp.(*streamReader).run+0x6f3		/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp/stream.go:311

2 @ 0x42cc8a 0x42754b 0x426ca9 0x4f88b8 0x4f8924 0x4fba78 0x515ece 0x514619 0x60944f 0x6090b7 0x7363e4 0x49adb9 0x49cced 0x45de01
#	0x426ca8	net.runtime_pollWait+0x58										/usr/local/go/src/runtime/netpoll.go:160
#	0x4f88b7	net.(*pollDesc).wait+0x37										/usr/local/go/src/net/fd_poll_runtime.go:73
#	0x4f8923	net.(*pollDesc).waitRead+0x33										/usr/local/go/src/net/fd_poll_runtime.go:78
#	0x4fba77	net.(*netFD).accept+0x237										/usr/local/go/src/net/fd_unix.go:419
#	0x515ecd	net.(*TCPListener).accept+0x2d										/usr/local/go/src/net/tcpsock_posix.go:132
#	0x514618	net.(*TCPListener).Accept+0x48										/usr/local/go/src/net/tcpsock.go:222
#	0x60944e	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/transport.(*limitListener).Accept+0x5e	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/transport/limit_listen.go:46
#	0x6090b6	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/transport.(*keepaliveListener).Accept+0x36	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/transport/keepalive_listener.go:49
#	0x7363e3	github.com/coreos/etcd/cmd/vendor/github.com/cockroachdb/cmux.(*cMux).Serve+0xa3			/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/cockroachdb/cmux/cmux.go:110
#	0x49adb8	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/embed.(*serveCtx).serve+0x238			/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/embed/serve.go:119
#	0x49ccec	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/embed.(*Etcd).serve.func2+0x6c			/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/embed/etcd.go:330

2 @ 0x42cc8a 0x42cd7e 0x40452f 0x4041f5 0x712068 0x45de01
#	0x712067	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb.RegisterKVHandlerFromEndpoint.func1.1+0x57	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.gw.go:586

2 @ 0x42cc8a 0x42cd7e 0x40452f 0x4041f5 0x7141b8 0x45de01
#	0x7141b7	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb.RegisterWatchHandlerFromEndpoint.func1.1+0x57	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.gw.go:783

2 @ 0x42cc8a 0x42cd7e 0x40452f 0x4041f5 0x714c68 0x45de01
#	0x714c67	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb.RegisterLeaseHandlerFromEndpoint.func1.1+0x57	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.gw.go:852

2 @ 0x42cc8a 0x42cd7e 0x40452f 0x4041f5 0x7168e8 0x45de01
#	0x7168e7	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb.RegisterClusterHandlerFromEndpoint.func1.1+0x57	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.gw.go:1017

2 @ 0x42cc8a 0x42cd7e 0x40452f 0x4041f5 0x718448 0x45de01
#	0x718447	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb.RegisterMaintenanceHandlerFromEndpoint.func1.1+0x57	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.gw.go:1182

2 @ 0x42cc8a 0x42cd7e 0x40452f 0x4041f5 0x71a6b8 0x45de01
#	0x71a6b7	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb.RegisterAuthHandlerFromEndpoint.func1.1+0x57	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.gw.go:1379

2 @ 0x42cc8a 0x42cd7e 0x40452f 0x404235 0x737e1f 0x56baae 0x49ce0f 0x45de01
#	0x737e1e	github.com/coreos/etcd/cmd/vendor/github.com/cockroachdb/cmux.(*muxListener).Accept+0x7e	<autogenerated>:9
#	0x56baad	net/http.(*Server).Serve+0x1cd									/usr/local/go/src/net/http/server.go:2273
#	0x49ce0e	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/embed.(*serveCtx).serve.func2+0x3e	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/embed/serve.go:87

2 @ 0x42cc8a 0x42cd7e 0x40452f 0x404235 0x737e1f 0x640c37 0x49cd7f 0x45de01
#	0x737e1e	github.com/coreos/etcd/cmd/vendor/github.com/cockroachdb/cmux.(*muxListener).Accept+0x7e	<autogenerated>:9
#	0x640c36	github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).Serve+0x186			/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:354
#	0x49cd7e	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/embed.(*serveCtx).serve.func1+0x3e	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/embed/serve.go:70

2 @ 0x42cc8a 0x42cd7e 0x40452f 0x404235 0xb553b4 0x45de01
#	0xb553b3	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/logutil.(*MergeLogger).outputLoop+0x93	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/logutil/merge_logger.go:174

2 @ 0x42cc8a 0x43c2b5 0x43b04c 0x606864 0x45de01
#	0x606863	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/fileutil.purgeFile.func1+0x853	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/fileutil/purge.go:70

2 @ 0x42cc8a 0x43c2b5 0x43b04c 0x783080 0x45de01
#	0x78307f	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp.monitorProbingStatus+0x54f	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp/probing_status.go:51

2 @ 0x42cc8a 0x43c2b5 0x43b04c 0x78dede 0x45de01
#	0x78dedd	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp.startPeer.func1+0x28d	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp/peer.go:156

2 @ 0x42cc8a 0x43c2b5 0x43b04c 0x78e17e 0x45de01
#	0x78e17d	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp.startPeer.func2+0x28d	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/rafthttp/peer.go:172

2 @ 0x42cc8a 0x43c2b5 0x43b04c 0x79bbe5 0x6b39ab 0x61d411 0x79dab5 0x644116 0x645476 0x64a9fb 0x45de01
#	0x79bbe4	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.(*watchServer).Watch+0x4d4	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/watch.go:139
#	0x6b39aa	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb._Watch_Watch_Handler+0xba	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.go:2459
#	0x61d410	github.com/coreos/etcd/cmd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.StreamServerInterceptor+0x100	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server.go:40
#	0x79dab4	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.newStreamInterceptor.func1+0x114	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/interceptor.go:93
#	0x644115	github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).processStreamingRPC+0x2b5			/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:723
#	0x645475	github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).handleStream+0xc35				/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:800
#	0x64a9fa	github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1+0xaa			/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:449

2 @ 0x42cc8a 0x43c2b5 0x43b04c 0x79d265 0x79e32d 0x45de01
#	0x79d264	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.(*serverWatchStream).sendLoop+0xd94	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/watch.go:255
#	0x79e32c	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.(*watchServer).Watch.func1+0x2c		/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/watch.go:125

2 @ 0x42cc8a 0x43c2b5 0x43b04c 0x79e1c2 0x45de01
#	0x79e1c1	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.monitorLeader.func1+0x391	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/interceptor.go:115

2 @ 0x42cc8a 0x43c2b5 0x43b04c 0x9b706a 0x9b7a59 0x744074 0x7441d8 0x63d53f 0x63dffd 0x648c68 0x61d5fb 0x6b4a72 0x797d98 0x79fddd 0x6b494b 0x61d411 0x79dab5 0x644116 0x645476 0x64a9fb 0x45de01
#	0x9b7069	github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport.(*recvBufferReader).Read+0x699			/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport/transport.go:140
#	0x9b7a58	github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport.(*Stream).Read+0x58					/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport/transport.go:325
#	0x744073	io.ReadAtLeast+0xa3													/usr/local/go/src/io/io.go:307
#	0x7441d7	io.ReadFull+0x57													/usr/local/go/src/io/io.go:325
#	0x63d53e	github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*parser).recvMsg+0x6e						/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/rpc_util.go:231
#	0x63dffc	github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.recv+0x4c							/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/rpc_util.go:315
#	0x648c67	github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*serverStream).RecvMsg+0xf7					/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/stream.go:516
#	0x61d5fa	github.com/coreos/etcd/cmd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.(*monitoredServerStream).RecvMsg+0x4a	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server.go:69
#	0x6b4a71	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb.(*leaseLeaseKeepAliveServer).Recv+0x61	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.go:2657
#	0x797d97	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.(*LeaseServer).LeaseKeepAlive+0x37	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/lease.go:65
#	0x79fddc	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.(*quotaLeaseServer).LeaseKeepAlive+0x5c	<autogenerated>:27
#	0x6b494a	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb._Lease_LeaseKeepAlive_Handler+0xba	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.go:2638
#	0x61d410	github.com/coreos/etcd/cmd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.StreamServerInterceptor+0x100		/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server.go:40
#	0x79dab4	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.newStreamInterceptor.func1+0x114		/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/interceptor.go:93
#	0x644115	github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).processStreamingRPC+0x2b5				/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:723
#	0x645475	github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).handleStream+0xc35					/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:800
#	0x64a9fa	github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1+0xaa				/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:449

2 @ 0x42cc8a 0x43c2b5 0x43b04c 0x9b706a 0x9b7a59 0x744074 0x7441d8 0x63d53f 0x63dffd 0x648c68 0x79ed3d 0x61d5fb 0x6b3ad2 0x79bc8c 0x79e37d 0x45de01
#	0x9b7069	github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport.(*recvBufferReader).Read+0x699			/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport/transport.go:140
#	0x9b7a58	github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport.(*Stream).Read+0x58					/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport/transport.go:325
#	0x744073	io.ReadAtLeast+0xa3													/usr/local/go/src/io/io.go:307
#	0x7441d7	io.ReadFull+0x57													/usr/local/go/src/io/io.go:325
#	0x63d53e	github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*parser).recvMsg+0x6e						/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/rpc_util.go:231
#	0x63dffc	github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.recv+0x4c							/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/rpc_util.go:315
#	0x648c67	github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*serverStream).RecvMsg+0xf7					/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/stream.go:516
#	0x79ed3c	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.(*serverStreamWithCtx).RecvMsg+0x5c	<autogenerated>:1
#	0x61d5fa	github.com/coreos/etcd/cmd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.(*monitoredServerStream).RecvMsg+0x4a	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server.go:69
#	0x6b3ad1	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb.(*watchWatchServer).Recv+0x61		/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.go:2478
#	0x79bc8b	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.(*serverWatchStream).recvLoop+0x4b	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/watch.go:155
#	0x79e37c	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.(*watchServer).Watch.func2+0x2c		/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/watch.go:135

2 @ 0x42cc8a 0x43c2b5 0x43b04c 0xb57ec5 0x45de01
#	0xb57ec4	github.com/coreos/etcd/cmd/vendor/github.com/xiang90/probing.(*prober).AddHTTP.func1+0x4b4	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/xiang90/probing/prober.go:55

1 @ 0x40e152 0x440257 0x922772 0x45de01
#	0x440256	os/signal.signal_recv+0x156	/usr/local/go/src/runtime/sigqueue.go:116
#	0x922771	os/signal.loop+0x21		/usr/local/go/src/os/signal/signal_unix.go:22

1 @ 0x42cc8a 0x42754b 0x426ca9 0x4f88b8 0x4f8924 0x4fba78 0x515ece 0x514619 0x60c037 0x56baae 0x49bbd2 0x49cc2d 0x45de01
#	0x426ca8	net.runtime_pollWait+0x58										/usr/local/go/src/runtime/netpoll.go:160
#	0x4f88b7	net.(*pollDesc).wait+0x37										/usr/local/go/src/net/fd_poll_runtime.go:73
#	0x4f8923	net.(*pollDesc).waitRead+0x33										/usr/local/go/src/net/fd_poll_runtime.go:78
#	0x4fba77	net.(*netFD).accept+0x237										/usr/local/go/src/net/fd_unix.go:419
#	0x515ecd	net.(*TCPListener).accept+0x2d										/usr/local/go/src/net/tcpsock_posix.go:132
#	0x514618	net.(*TCPListener).Accept+0x48										/usr/local/go/src/net/tcpsock.go:222
#	0x60c036	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/transport.(*rwTimeoutListener).Accept+0x36	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/transport/timeout_listener.go:49
#	0x56baad	net/http.(*Server).Serve+0x1cd										/usr/local/go/src/net/http/server.go:2273
#	0x49bbd1	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/embed.servePeerHTTP+0x161			/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/embed/serve.go:142
#	0x49cc2c	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/embed.(*Etcd).serve.func1+0x4c			/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/embed/etcd.go:317

1 @ 0x42cc8a 0x42cd7e 0x40452f 0x4041f5 0x6089c1 0x45de01
#	0x6089c0	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/osutil.HandleInterrupts.func1+0x70	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/osutil/interrupt_unix.go:51

1 @ 0x42cc8a 0x42cd7e 0x43ce8d 0x43cb20 0x74a4f0 0x74b4d1 0x8b7cf3 0x8b8f37 0x5d588a 0x6017a3 0x5fdbd3 0x5d10e1 0x601633 0x5d6d82 0x5ecd6e 0x5ecaa1 0x5e95e7 0x5e7884 0x5f9db0 0x91c20d 0x45de01
#	0x43cb1f	sync.runtime_Semacquire+0x2f											/usr/local/go/src/runtime/sema.go:47
#	0x74a4ef	sync.(*Mutex).Lock+0xcf												/usr/local/go/src/sync/mutex.go:85
#	0x74b4d0	sync.(*RWMutex).Lock+0x30											/usr/local/go/src/sync/rwmutex.go:86
#	0x8b7cf2	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth.(*authStore).assignSimpleTokenToUser+0x52		/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth/simple_token.go:119
#	0x8b8f36	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth.(*authStore).Authenticate+0x406			/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth/store.go:289
#	0x5d5889	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*applierV3backend).Authenticate+0x289	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/apply.go:622
#	0x6017a2	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*quotaApplierV3).Authenticate+0x52		<autogenerated>:174
#	0x5fdbd2	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*authApplierV3).Authenticate+0x52		<autogenerated>:66
#	0x5d10e0	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*applierV3backend).Apply+0x700		/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/apply.go:118
#	0x601632	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*quotaApplierV3).Apply+0x52		<autogenerated>:171
#	0x5d6d81	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*authApplierV3).Apply+0xe1			/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/apply_auth.go:55
#	0x5ecd6d	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).applyEntryNormal+0x28d	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:1323
#	0x5ecaa0	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).apply+0x2c0			/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:1252
#	0x5e95e6	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).applyEntries+0xc6		/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:891
#	0x5e7883	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).applyAll+0xc3			/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:750
#	0x5f9daf	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).run.func6+0x3f		/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:714
#	0x91c20c	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/schedule.(*fifo).run+0xdc				/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/schedule/schedule.go:160

1 @ 0x42cc8a 0x42cd7e 0x43ce8d 0x43cb20 0x74a4f0 0x8dbc4d 0x8d9a1b 0x45de01
#	0x43cb1f	sync.runtime_Semacquire+0x2f									/usr/local/go/src/runtime/sema.go:47
#	0x74a4ef	sync.(*Mutex).Lock+0xcf										/usr/local/go/src/sync/mutex.go:85
#	0x8dbc4c	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/mvcc/backend.(*batchTx).Commit+0x2c	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/mvcc/backend/batch_tx.go:138
#	0x8d9a1a	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/mvcc/backend.(*backend).run+0x13a	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/mvcc/backend/backend.go:199

1 @ 0x42cc8a 0x42cd7e 0x43ce8d 0x43cb20 0x74b538 0x8bffac 0x8b7998 0x45de01
#	0x43cb1f	sync.runtime_Semacquire+0x2f									/usr/local/go/src/runtime/sema.go:47
#	0x74b537	sync.(*RWMutex).Lock+0x97									/usr/local/go/src/sync/rwmutex.go:91
#	0x8bffab	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth.newDeleterFunc.func1+0x5b		/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth/store.go:182
#	0x8b7997	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth.(*simpleTokenTTLKeeper).run+0x557	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth/simple_token.go:91

1 @ 0x42cc8a 0x43c2b5 0x43b04c 0x455683 0x45de01
#	0x42cc89	runtime.gopark+0x139		/usr/local/go/src/runtime/proc.go:259
#	0x43c2b4	runtime.selectgoImpl+0x1234	/usr/local/go/src/runtime/select.go:423
#	0x43b04b	runtime.selectgo+0x1b		/usr/local/go/src/runtime/select.go:238
#	0x455682	runtime.ensureSigM.func1+0x2f2	/usr/local/go/src/runtime/signal1_unix.go:304

1 @ 0x42cc8a 0x43c2b5 0x43b04c 0x462ca0 0x468861 0x401014 0x42c824 0x45de01
#	0x462c9f	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain.startEtcdOrProxyV2+0xc8f	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain/etcd.go:171
#	0x468860	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain.Main+0x60			/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain/main.go:39
#	0x401013	main.main+0x13											/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/etcd/main.go:28
#	0x42c823	runtime.main+0x1f3										/usr/local/go/src/runtime/proc.go:183

1 @ 0x42cc8a 0x43c2b5 0x43b04c 0x5ddbfa 0x5f9861 0x5fb086 0x45de01
#	0x5ddbf9	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.monitorFileDescriptor+0x1b9		/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/metrics.go:89
#	0x5f9860	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).Start.func2+0x30	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:515
#	0x5fb085	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).goAttach.func1+0x55	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:1638

1 @ 0x42cc8a 0x43c2b5 0x43b04c 0x5e653a 0x5f981a 0x5fb086 0x45de01
#	0x5e6539	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).purgeFile+0x2c9										/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:553
#	0x5f9819	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).(github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.purgeFile)-fm+0x29	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:514
#	0x5fb085	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).goAttach.func1+0x55										/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:1638

1 @ 0x42cc8a 0x43c2b5 0x43b04c 0x5e74b7 0x45de01
#	0x5e74b6	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).run+0x946	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:712

1 @ 0x42cc8a 0x43c2b5 0x43b04c 0x5ee19b 0x5f98aa 0x5fb086 0x45de01
#	0x5ee19a	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).monitorVersions+0x49a										/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:1488
#	0x5f98a9	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).(github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.monitorVersions)-fm+0x29	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:516
#	0x5fb085	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).goAttach.func1+0x55										/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:1638

1 @ 0x42cc8a 0x43c2b5 0x43b04c 0x5f7e41 0x5f98ea 0x5fb086 0x45de01
#	0x5f7e40	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).linearizableReadLoop+0x870										/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/v3_server.go:773
#	0x5f98e9	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).(github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.linearizableReadLoop)-fm+0x29	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:517
#	0x5fb085	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).goAttach.func1+0x55											/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:1638

1 @ 0x42cc8a 0x43c2b5 0x43b04c 0x5f941f 0x45de01
#	0x5f941e	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*raftNode).start.func1+0xdbe	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/raft.go:145

1 @ 0x42cc8a 0x43c2b5 0x43b04c 0x7af0e7 0x45de01
#	0x7af0e6	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/wal.(*filePipeline).run+0x196	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/wal/file_pipeline.go:89

1 @ 0x42cc8a 0x43c2b5 0x43b04c 0x8809f4 0x45de01
#	0x8809f3	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/lease.(*lessor).runLoop+0x193	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/lease/lessor.go:447

1 @ 0x42cc8a 0x43c2b5 0x43b04c 0x88d6e1 0x45de01
#	0x88d6e0	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/mvcc.(*watchableStore).syncWatchersLoop+0x200	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/mvcc/watchable_store.go:281

1 @ 0x42cc8a 0x43c2b5 0x43b04c 0x88d8ff 0x45de01
#	0x88d8fe	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/mvcc.(*watchableStore).syncVictimsLoop+0x1ce	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/mvcc/watchable_store.go:307

1 @ 0x42cc8a 0x43c2b5 0x43b04c 0x8e4e9b 0x45de01
#	0x8e4e9a	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft.(*node).run+0x107a	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft/node.go:309

1 @ 0x42cc8a 0x43c2b5 0x43b04c 0x91c457 0x45de01
#	0x91c456	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/schedule.(*fifo).run+0x326	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/schedule/schedule.go:146

1 @ 0xb71f3f 0xb71d40 0xb6ea61 0x794116 0x794709 0x56ab3f 0x56b62d 0x567fd7 0x45de01
#	0xb71f3e	runtime/pprof.writeRuntimeProfile+0x9e		/usr/local/go/src/runtime/pprof/pprof.go:614
#	0xb71d3f	runtime/pprof.writeGoroutine+0x9f		/usr/local/go/src/runtime/pprof/pprof.go:576
#	0xb6ea60	runtime/pprof.(*Profile).WriteTo+0x340		/usr/local/go/src/runtime/pprof/pprof.go:298
#	0x794115	net/http/pprof.handler.ServeHTTP+0x1a5		/usr/local/go/src/net/http/pprof/pprof.go:209
#	0x794708	net/http/pprof.(*handler).ServeHTTP+0x78	<autogenerated>:1
#	0x56ab3e	net/http.(*ServeMux).ServeHTTP+0x7e		/usr/local/go/src/net/http/server.go:2022
#	0x56b62c	net/http.serverHandler.ServeHTTP+0x7c		/usr/local/go/src/net/http/server.go:2202
#	0x567fd6	net/http.(*conn).serve+0x4b6			/usr/local/go/src/net/http/server.go:1579

xiang90 · 2017-03-13T21:52:44Z

@davissp14 Seems like a but in Auth layer. Can you try to disable auth to see if it happens again?

heyitsanthony · 2017-03-13T22:01:42Z

There's a deadlock in simpleTokenTTLKeeper. assignSimpleTokenToUser acquires as.simpleTokensMu and posts to addSimpleTokenCh (suppose that the channel is full so it blocks). If the goroutine simpleTokenTTLKeeper.run happens to hit <-tokenTicker.C it will try to acquire simpleTokensMu while calling deleteTokenFunc. Since only run can drain addSimpleTokenCh, the lock is never released.

davissp14 · 2017-03-13T22:11:24Z

@xiang90 Confirmed it works just fine when disabling Auth.

Just use the mutex instead. Fixes etcd-io#7471

heyitsanthony mentioned this issue Mar 10, 2017

auth: nil check AuthInfo when checking admin permissions #7479

Merged

xiang90 added type/bug component/etcdserver/auth labels Mar 13, 2017

heyitsanthony pushed a commit to heyitsanthony/etcd that referenced this issue Mar 13, 2017

auth: get rid of deadlocking channel passing scheme in simpleTokenTTL

b5bfd98

Just use the mutex instead. Fixes etcd-io#7471

heyitsanthony mentioned this issue Mar 13, 2017

auth: get rid of deadlocking channel passing scheme in simpleTokenTTL #7492

Merged

heyitsanthony pushed a commit to heyitsanthony/etcd that referenced this issue Mar 13, 2017

auth: get rid of deadlocking channel passing scheme in simpleTokenTTL

d0dc707

Just use the mutex instead. Fixes etcd-io#7471

heyitsanthony pushed a commit to heyitsanthony/etcd that referenced this issue Mar 14, 2017

auth: get rid of deadlocking channel passing scheme in simpleTokenTTL

1b1fabe

Just use the mutex instead. Fixes etcd-io#7471

mitake closed this as completed in #7492 Mar 14, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Benchmarking is causing cluster to become unresponsive #7471

Benchmarking is causing cluster to become unresponsive #7471

davissp14 commented Mar 9, 2017 •

edited

Loading

gyuho commented Mar 10, 2017 •

edited

Loading

xiang90 commented Mar 10, 2017

davissp14 commented Mar 10, 2017 •

edited

Loading

davissp14 commented Mar 10, 2017 •

edited

Loading

gyuho commented Mar 10, 2017

heyitsanthony commented Mar 10, 2017

davissp14 commented Mar 10, 2017

davissp14 commented Mar 10, 2017 •

edited

Loading

davissp14 commented Mar 13, 2017 •

edited

Loading

xiang90 commented Mar 13, 2017

heyitsanthony commented Mar 13, 2017

davissp14 commented Mar 13, 2017

Benchmarking is causing cluster to become unresponsive #7471

Benchmarking is causing cluster to become unresponsive #7471

Comments

davissp14 commented Mar 9, 2017 • edited Loading

gyuho commented Mar 10, 2017 • edited Loading

xiang90 commented Mar 10, 2017

davissp14 commented Mar 10, 2017 • edited Loading

davissp14 commented Mar 10, 2017 • edited Loading

gyuho commented Mar 10, 2017

heyitsanthony commented Mar 10, 2017

davissp14 commented Mar 10, 2017

davissp14 commented Mar 10, 2017 • edited Loading

davissp14 commented Mar 13, 2017 • edited Loading

xiang90 commented Mar 13, 2017

heyitsanthony commented Mar 13, 2017

davissp14 commented Mar 13, 2017

davissp14 commented Mar 9, 2017 •

edited

Loading

gyuho commented Mar 10, 2017 •

edited

Loading

davissp14 commented Mar 10, 2017 •

edited

Loading

davissp14 commented Mar 10, 2017 •

edited

Loading

davissp14 commented Mar 10, 2017 •

edited

Loading

davissp14 commented Mar 13, 2017 •

edited

Loading