Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

multiraft: TestRemoveLeader deadlocks in multi cpu #2639

Closed
tamird opened this issue Sep 23, 2015 · 2 comments
Closed

multiraft: TestRemoveLeader deadlocks in multi cpu #2639

tamird opened this issue Sep 23, 2015 · 2 comments
Assignees
@bdarnell
Labels
C-test-failure Broken test (automatically or manually discovered).

Comments

@tamird
Copy link
Contributor

tamird commented Sep 23, 2015

discovered in #1839

https://circleci.com/gh/cockroachdb/cockroach/7882
@tamird tamird added C-test-failure Broken test (automatically or manually discovered). multi-cpu labels Sep 23, 2015
@tamird
Copy link
Contributor Author

tamird commented Oct 6, 2015

Haven't seen in a while, closing.

@tamird tamird closed this as completed Oct 6, 2015
@tamird
Copy link
Contributor Author

tamird commented Nov 5, 2015

Reopening, I'm able to reproduce this locally in multi-cpu:

$ time rt testrace multiraft RemoveLeader
go test -tags ''  -race -i ./multiraft
go test -tags ''  -race -run . ./multiraft -timeout 5m -c
..................I1105 07:03:43.695858 29707 multiraft/multiraft_test.go:465  adding node 3
I1105 07:03:43.739019 29707 multiraft/multiraft_test.go:477  removing node 0
panic: test timed out after 2m0s

goroutine 286908 [running]:
testing.startAlarm.func1()
    /Users/tamird/src/go1.5/src/testing/testing.go:703 +0x16b
created by time.goFunc
    /Users/tamird/src/go1.5/src/time/sleep.go:129 +0x6e

goroutine 1 [chan receive, 1 minutes]:
testing.RunTests(0x49178c0, 0x4b0f960, 0xb, 0xb, 0x1)
    /Users/tamird/src/go1.5/src/testing/testing.go:562 +0xafa
testing.(*M).Run(0xc820211f10, 0x0)
    /Users/tamird/src/go1.5/src/testing/testing.go:494 +0xe5
github.com/cockroachdb/cockroach/util/leaktest.TestMainWithLeakCheck(0xc820211f10)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/leaktest/leaktest.go:34 +0x2f
github.com/cockroachdb/cockroach/multiraft_test.TestMain(0xc820211f10)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/multiraft/main_test.go:29 +0x2f
main.main()
    github.com/cockroachdb/cockroach/multiraft/_test/_testmain.go:74 +0x20a

goroutine 17 [syscall, 1 minutes, locked to thread]:
runtime.goexit()
    /Users/tamird/src/go1.5/src/runtime/asm_amd64.s:1696 +0x1

goroutine 5 [chan receive]:
github.com/cockroachdb/cockroach/util/log.(*loggingT).flushDaemon(0x54ab920)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/log/clog.go:1029 +0x76
created by github.com/cockroachdb/cockroach/util/log.init.1
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/log/clog.go:610 +0x117

goroutine 7 [select]:
github.com/cockroachdb/cockroach/multiraft.(*writeTask).start.func1()
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/multiraft/storage.go:199 +0x1031
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820132600, 0xc82000f900)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x60
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x70

goroutine 66 [select]:
github.com/cockroachdb/cockroach/multiraft.(*writeTask).start.func1()
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/multiraft/storage.go:199 +0x1031
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820132600, 0xc82015c480)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x60
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x70

goroutine 36 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/multiraft.TestRemoveLeader(0xc82019e000)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/multiraft/multiraft_test.go:485 +0x13c8
testing.tRunner(0xc82019e000, 0x4b0fa20)
    /Users/tamird/src/go1.5/src/testing/testing.go:456 +0xdd
created by testing.RunTests
    /Users/tamird/src/go1.5/src/testing/testing.go:561 +0xaa4

goroutine 37 [select]:
github.com/coreos/etcd/raft.(*multiNode).run(0xc820132720)
    /Users/tamird/src/go/src/github.com/coreos/etcd/raft/multinode.go:195 +0x2c7d
created by github.com/coreos/etcd/raft.StartMultiNode
    /Users/tamird/src/go/src/github.com/coreos/etcd/raft/multinode.go:71 +0x338

goroutine 38 [IO wait, 1 minutes]:
net.runtime_pollWait(0x5fa5800, 0x72, 0x0)
    /Users/tamird/src/go1.5/src/runtime/netpoll.go:157 +0x63
net.(*pollDesc).Wait(0xc820188b50, 0x72, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/fd_poll_runtime.go:73 +0x56
net.(*pollDesc).WaitRead(0xc820188b50, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/fd_poll_runtime.go:78 +0x44
net.(*netFD).accept(0xc820188af0, 0x0, 0x69601c8, 0xc82015c5e0)
    /Users/tamird/src/go1.5/src/net/fd_unix.go:408 +0x2f6
net.(*TCPListener).AcceptTCP(0xc8201201f8, 0xc8201aee20, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/tcpsock_posix.go:254 +0x77
net.(*TCPListener).Accept(0xc8201201f8, 0x0, 0x0, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/tcpsock_posix.go:264 +0x4b
net/http.(*Server).Serve(0xc820132de0, 0x5fa4858, 0xc8201201f8, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/http/server.go:1887 +0xc4
github.com/cockroachdb/cockroach/rpc.(*Server).Serve.func2()
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/server.go:264 +0x78
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820132600, 0xc82011f060)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x60
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x70

goroutine 39 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/rpc.(*Server).Serve.func3()
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/server.go:270 +0x87
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820132600, 0xc820113500)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x60
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x70

goroutine 40 [select, 1 minutes]:
github.com/cockroachdb/cockroach/multiraft.(*eventDemux).start.func1()
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/multiraft/events_test.go:49 +0x5f2
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820132600, 0xc82011f0a0)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x60
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x70

goroutine 41 [select]:
github.com/coreos/etcd/raft.(*multiNode).run(0xc8201330e0)
    /Users/tamird/src/go/src/github.com/coreos/etcd/raft/multinode.go:195 +0x2c7d
created by github.com/coreos/etcd/raft.StartMultiNode
    /Users/tamird/src/go/src/github.com/coreos/etcd/raft/multinode.go:71 +0x338

goroutine 42 [IO wait, 1 minutes]:
net.runtime_pollWait(0x5fa5740, 0x72, 0x0)
    /Users/tamird/src/go1.5/src/runtime/netpoll.go:157 +0x63
net.(*pollDesc).Wait(0xc820188c30, 0x72, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/fd_poll_runtime.go:73 +0x56
net.(*pollDesc).WaitRead(0xc820188c30, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/fd_poll_runtime.go:78 +0x44
net.(*netFD).accept(0xc820188bd0, 0x0, 0x69601c8, 0xc82011f800)
    /Users/tamird/src/go1.5/src/net/fd_unix.go:408 +0x2f6
net.(*TCPListener).AcceptTCP(0xc820120238, 0xc8201b0e20, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/tcpsock_posix.go:254 +0x77
net.(*TCPListener).Accept(0xc820120238, 0x0, 0x0, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/tcpsock_posix.go:264 +0x4b
net/http.(*Server).Serve(0xc8201337a0, 0x5fa4858, 0xc820120238, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/http/server.go:1887 +0xc4
github.com/cockroachdb/cockroach/rpc.(*Server).Serve.func2()
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/server.go:264 +0x78
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820132600, 0xc82011f160)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x60
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x70

goroutine 43 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/rpc.(*Server).Serve.func3()
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/server.go:270 +0x87
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820132600, 0xc820113690)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x60
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x70

goroutine 44 [select, 1 minutes]:
github.com/cockroachdb/cockroach/multiraft.(*eventDemux).start.func1()
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/multiraft/events_test.go:49 +0x5f2
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820132600, 0xc82011f1a0)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x60
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x70

goroutine 45 [select]:
github.com/coreos/etcd/raft.(*multiNode).run(0xc820133aa0)
    /Users/tamird/src/go/src/github.com/coreos/etcd/raft/multinode.go:195 +0x2c7d
created by github.com/coreos/etcd/raft.StartMultiNode
    /Users/tamird/src/go/src/github.com/coreos/etcd/raft/multinode.go:71 +0x338

goroutine 46 [IO wait, 1 minutes]:
net.runtime_pollWait(0x5fa5680, 0x72, 0x0)
    /Users/tamird/src/go1.5/src/runtime/netpoll.go:157 +0x63
net.(*pollDesc).Wait(0xc820188d10, 0x72, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/fd_poll_runtime.go:73 +0x56
net.(*pollDesc).WaitRead(0xc820188d10, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/fd_poll_runtime.go:78 +0x44
net.(*netFD).accept(0xc820188cb0, 0x0, 0x69601c8, 0xc8200bed80)
    /Users/tamird/src/go1.5/src/net/fd_unix.go:408 +0x2f6
net.(*TCPListener).AcceptTCP(0xc820120258, 0xc8201aae20, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/tcpsock_posix.go:254 +0x77
net.(*TCPListener).Accept(0xc820120258, 0x0, 0x0, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/tcpsock_posix.go:264 +0x4b
net/http.(*Server).Serve(0xc8201ce180, 0x5fa4858, 0xc820120258, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/http/server.go:1887 +0xc4
github.com/cockroachdb/cockroach/rpc.(*Server).Serve.func2()
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/server.go:264 +0x78
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820132600, 0xc82011f260)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x60
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x70

goroutine 47 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/rpc.(*Server).Serve.func3()
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/server.go:270 +0x87
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820132600, 0xc820113860)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x60
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x70

goroutine 48 [select, 1 minutes]:
github.com/cockroachdb/cockroach/multiraft.(*eventDemux).start.func1()
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/multiraft/events_test.go:49 +0x5f2
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820132600, 0xc82011f2a0)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x60
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x70

goroutine 49 [select]:
github.com/coreos/etcd/raft.(*multiNode).run(0xc8201ce480)
    /Users/tamird/src/go/src/github.com/coreos/etcd/raft/multinode.go:195 +0x2c7d
created by github.com/coreos/etcd/raft.StartMultiNode
    /Users/tamird/src/go/src/github.com/coreos/etcd/raft/multinode.go:71 +0x338

goroutine 50 [IO wait, 1 minutes]:
net.runtime_pollWait(0x5fa55c0, 0x72, 0x0)
    /Users/tamird/src/go1.5/src/runtime/netpoll.go:157 +0x63
net.(*pollDesc).Wait(0xc820188df0, 0x72, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/fd_poll_runtime.go:73 +0x56
net.(*pollDesc).WaitRead(0xc820188df0, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/fd_poll_runtime.go:78 +0x44
net.(*netFD).accept(0xc820188d90, 0x0, 0x69601c8, 0xc82015cb80)
    /Users/tamird/src/go1.5/src/net/fd_unix.go:408 +0x2f6
net.(*TCPListener).AcceptTCP(0xc820120278, 0xc8201abe20, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/tcpsock_posix.go:254 +0x77
net.(*TCPListener).Accept(0xc820120278, 0x0, 0x0, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/tcpsock_posix.go:264 +0x4b
net/http.(*Server).Serve(0xc8201ceb40, 0x5fa4858, 0xc820120278, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/http/server.go:1887 +0xc4
github.com/cockroachdb/cockroach/rpc.(*Server).Serve.func2()
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/server.go:264 +0x78
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820132600, 0xc82011f3e0)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x60
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x70

goroutine 51 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/rpc.(*Server).Serve.func3()
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/server.go:270 +0x87
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820132600, 0xc8201139f0)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x60
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x70

goroutine 52 [select, 1 minutes]:
github.com/cockroachdb/cockroach/multiraft.(*eventDemux).start.func1()
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/multiraft/events_test.go:49 +0x5f2
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820132600, 0xc82011f420)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x60
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x70

goroutine 53 [select]:
github.com/coreos/etcd/raft.(*multiNode).run(0xc8201cee40)
    /Users/tamird/src/go/src/github.com/coreos/etcd/raft/multinode.go:195 +0x2c7d
created by github.com/coreos/etcd/raft.StartMultiNode
    /Users/tamird/src/go/src/github.com/coreos/etcd/raft/multinode.go:71 +0x338

goroutine 54 [IO wait, 1 minutes]:
net.runtime_pollWait(0x5fa5500, 0x72, 0x0)
    /Users/tamird/src/go1.5/src/runtime/netpoll.go:157 +0x63
net.(*pollDesc).Wait(0xc820188ed0, 0x72, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/fd_poll_runtime.go:73 +0x56
net.(*pollDesc).WaitRead(0xc820188ed0, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/fd_poll_runtime.go:78 +0x44
net.(*netFD).accept(0xc820188e70, 0x0, 0x69601c8, 0xc8200be9e0)
    /Users/tamird/src/go1.5/src/net/fd_unix.go:408 +0x2f6
net.(*TCPListener).AcceptTCP(0xc820120298, 0x408bd20, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/tcpsock_posix.go:254 +0x77
net.(*TCPListener).Accept(0xc820120298, 0x0, 0x0, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/tcpsock_posix.go:264 +0x4b
net/http.(*Server).Serve(0xc8201cf500, 0x5fa4858, 0xc820120298, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/http/server.go:1887 +0xc4
github.com/cockroachdb/cockroach/rpc.(*Server).Serve.func2()
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/server.go:264 +0x78
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820132600, 0xc82011f4e0)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x60
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x70

goroutine 55 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/rpc.(*Server).Serve.func3()
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/server.go:270 +0x87
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820132600, 0xc820113b80)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x60
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x70

goroutine 56 [select, 1 minutes]:
github.com/cockroachdb/cockroach/multiraft.(*eventDemux).start.func1()
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/multiraft/events_test.go:49 +0x5f2
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820132600, 0xc82011f520)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x60
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x70

goroutine 57 [select]:
github.com/coreos/etcd/raft.(*multiNode).run(0xc8201cf800)
    /Users/tamird/src/go/src/github.com/coreos/etcd/raft/multinode.go:195 +0x2c7d
created by github.com/coreos/etcd/raft.StartMultiNode
    /Users/tamird/src/go/src/github.com/coreos/etcd/raft/multinode.go:71 +0x338

goroutine 58 [IO wait, 1 minutes]:
net.runtime_pollWait(0x5fa5440, 0x72, 0x0)
    /Users/tamird/src/go1.5/src/runtime/netpoll.go:157 +0x63
net.(*pollDesc).Wait(0xc820188fb0, 0x72, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/fd_poll_runtime.go:73 +0x56
net.(*pollDesc).WaitRead(0xc820188fb0, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/fd_poll_runtime.go:78 +0x44
net.(*netFD).accept(0xc820188f50, 0x0, 0x69601c8, 0xc8200bea00)
    /Users/tamird/src/go1.5/src/net/fd_unix.go:408 +0x2f6
net.(*TCPListener).AcceptTCP(0xc8201202b8, 0x408bd20, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/tcpsock_posix.go:254 +0x77
net.(*TCPListener).Accept(0xc8201202b8, 0x0, 0x0, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/tcpsock_posix.go:264 +0x4b
net/http.(*Server).Serve(0xc8201cfec0, 0x5fa4858, 0xc8201202b8, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/http/server.go:1887 +0xc4
github.com/cockroachdb/cockroach/rpc.(*Server).Serve.func2()
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/server.go:264 +0x78
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820132600, 0xc82011f5e0)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x60
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x70

goroutine 59 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/rpc.(*Server).Serve.func3()
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/server.go:270 +0x87
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820132600, 0xc820113d10)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x60
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x70

goroutine 60 [select, 1 minutes]:
github.com/cockroachdb/cockroach/multiraft.(*eventDemux).start.func1()
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/multiraft/events_test.go:49 +0x5f2
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820132600, 0xc82011f620)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x60
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x70

goroutine 61 [select]:
github.com/cockroachdb/cockroach/multiraft.(*state).start.func1()
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/multiraft/multiraft.go:570 +0x1cf1
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820132600, 0xc820113d30)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x60
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x70

goroutine 62 [select]:
github.com/cockroachdb/cockroach/multiraft.(*state).start.func1()
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/multiraft/multiraft.go:570 +0x1cf1
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820132600, 0xc820113d40)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x60
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x70

goroutine 63 [select]:
github.com/cockroachdb/cockroach/multiraft.(*state).start.func1()
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/multiraft/multiraft.go:570 +0x1cf1
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820132600, 0xc820113d50)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x60
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x70

goroutine 20 [select]:
github.com/cockroachdb/cockroach/multiraft.(*writeTask).start.func1()
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/multiraft/storage.go:199 +0x1031
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820132600, 0xc8200bea20)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x60
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x70

goroutine 64 [select]:
github.com/cockroachdb/cockroach/multiraft.(*state).start.func1()
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/multiraft/multiraft.go:570 +0x1cf1
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820132600, 0xc820113d60)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x60
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x70

goroutine 21 [select]:
github.com/cockroachdb/cockroach/multiraft.(*writeTask).start.func1()
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/multiraft/storage.go:199 +0x1031
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820132600, 0xc8200bea40)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x60
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x70

goroutine 65 [select]:
github.com/cockroachdb/cockroach/multiraft.(*state).start.func1()
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/multiraft/multiraft.go:570 +0x1cf1
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820132600, 0xc820113d70)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x60
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x70

goroutine 82 [select]:
github.com/cockroachdb/cockroach/multiraft.(*state).start.func1()
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/multiraft/multiraft.go:570 +0x1cf1
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820132600, 0xc820113d80)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x60
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x70

goroutine 22 [select, 1 minutes]:
github.com/cockroachdb/cockroach/multiraft.(*writeTask).start.func1()
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/multiraft/storage.go:199 +0x1031
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820132600, 0xc8200bea60)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x60
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x70

goroutine 83 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/multiraft.TestRemoveLeader.func1(0xc820122a80, 0x0)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/multiraft/multiraft_test.go:429 +0x11d
created by github.com/cockroachdb/cockroach/multiraft.TestRemoveLeader
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/multiraft/multiraft_test.go:435 +0x409

goroutine 67 [select, 1 minutes]:
github.com/cockroachdb/cockroach/multiraft.(*writeTask).start.func1()
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/multiraft/storage.go:199 +0x1031
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820132600, 0xc82015c4a0)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x60
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x70

goroutine 84 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/multiraft.TestRemoveLeader.func1(0xc820122a80, 0x1)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/multiraft/multiraft_test.go:429 +0x11d
created by github.com/cockroachdb/cockroach/multiraft.TestRemoveLeader
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/multiraft/multiraft_test.go:435 +0x409

goroutine 85 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/multiraft.TestRemoveLeader.func1(0xc820122a80, 0x2)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/multiraft/multiraft_test.go:429 +0x11d
created by github.com/cockroachdb/cockroach/multiraft.TestRemoveLeader
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/multiraft/multiraft_test.go:435 +0x409

goroutine 86 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/multiraft.TestRemoveLeader.func1(0xc820122a80, 0x3)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/multiraft/multiraft_test.go:429 +0x11d
created by github.com/cockroachdb/cockroach/multiraft.TestRemoveLeader
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/multiraft/multiraft_test.go:435 +0x409

goroutine 87 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/multiraft.TestRemoveLeader.func1(0xc820122a80, 0x4)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/multiraft/multiraft_test.go:429 +0x11d
created by github.com/cockroachdb/cockroach/multiraft.TestRemoveLeader
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/multiraft/multiraft_test.go:435 +0x409

goroutine 88 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/multiraft.TestRemoveLeader.func1(0xc820122a80, 0x5)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/multiraft/multiraft_test.go:429 +0x11d
created by github.com/cockroachdb/cockroach/multiraft.TestRemoveLeader
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/multiraft/multiraft_test.go:435 +0x409

goroutine 89 [runnable]:
github.com/cockroachdb/cockroach/multiraft.TestRemoveLeader.func2()
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/multiraft/multiraft_test.go:446 +0x1f9
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820132600, 0xc82011f640)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:88 +0x60
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x70

goroutine 90 [IO wait]:
net.runtime_pollWait(0x5fa52c0, 0x72, 0x0)
    /Users/tamird/src/go1.5/src/runtime/netpoll.go:157 +0x63
net.(*pollDesc).Wait(0xc820189170, 0x72, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/fd_poll_runtime.go:73 +0x56
net.(*pollDesc).WaitRead(0xc820189170, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/fd_poll_runtime.go:78 +0x44
net.(*netFD).Read(0xc820189110, 0xc820257000, 0x1000, 0x1000, 0x0, 0x5aa2050, 0xc8200121d0)
    /Users/tamird/src/go1.5/src/net/fd_unix.go:232 +0x27b
net.(*conn).Read(0xc8201202d0, 0xc820257000, 0x1000, 0x1000, 0xc820265280, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/net.go:172 +0x121
net.(*TCPConn).Read(0xc8201202d0, 0xc820257000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    <autogenerated>:74 +0x7d
bufio.(*Reader).fill(0xc820204300)
    /Users/tamird/src/go1.5/src/bufio/bufio.go:97 +0x365
bufio.(*Reader).ReadByte(0xc820204300, 0x403b1a5, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/bufio/bufio.go:229 +0x153
encoding/binary.ReadUvarint(0x5fa9d80, 0xc820204300, 0x0, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/encoding/binary/varint.go:110 +0x60
github.com/cockroachdb/cockroach/rpc/codec.(*baseConn).recvProto(0xc8201fbc80, 0x5fa9d48, 0xc8201fbdd8, 0x0, 0x4916ef8, 0x0, 0x0)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/codec/conn.go:88 +0x9e
github.com/cockroachdb/cockroach/rpc/codec.(*serverCodec).readRequestHeader(0xc8201fbc80, 0xc820204300, 0xc8201fbdd8, 0x0, 0x0)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/codec/server.go:173 +0x91
github.com/cockroachdb/cockroach/rpc/codec.(*serverCodec).ReadRequestHeader(0xc8201fbc80, 0xc820566ca0, 0x0, 0x0)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/codec/server.go:60 +0x8c
github.com/cockroachdb/cockroach/rpc.(*Server).readRequest(0xc820122b80, 0x5fa9d08, 0xc8201fbc80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/server.go:403 +0xe7
github.com/cockroachdb/cockroach/rpc.(*Server).readRequests(0xc820122b80, 0x5fa9d08, 0xc8201fbc80, 0xc82011f840, 0xc820204360)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/server.go:361 +0xfa
github.com/cockroachdb/cockroach/rpc.(*Server).ServeHTTP(0xc820122b80, 0x5fa9c48, 0xc8201286e0, 0xc820130540)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/server.go:198 +0xace
net/http.serverHandler.ServeHTTP(0xc8201337a0, 0x5fa9c48, 0xc8201286e0, 0xc820130540)
    /Users/tamird/src/go1.5/src/net/http/server.go:1862 +0x207
net/http.(*conn).serve(0xc820128630)
    /Users/tamird/src/go1.5/src/net/http/server.go:1361 +0x117d
created by net/http.(*Server).Serve
    /Users/tamird/src/go1.5/src/net/http/server.go:1910 +0x465

goroutine 91 [chan receive]:
github.com/cockroachdb/cockroach/rpc.(*Server).sendResponses(0xc820122b80, 0x5fa9d08, 0xc8201fbc80, 0xc820204360)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/server.go:426 +0xb5
github.com/cockroachdb/cockroach/rpc.(*Server).ServeHTTP.func1(0xc820122b80, 0x5fa9d08, 0xc8201fbc80, 0xc820204360, 0xc820250050)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/server.go:195 +0x4d
created by github.com/cockroachdb/cockroach/rpc.(*Server).ServeHTTP
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/server.go:197 +0xa8c

goroutine 68 [IO wait]:
net.runtime_pollWait(0x5fa5380, 0x72, 0x0)
    /Users/tamird/src/go1.5/src/runtime/netpoll.go:157 +0x63
net.(*pollDesc).Wait(0xc82024c060, 0x72, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/fd_poll_runtime.go:73 +0x56
net.(*pollDesc).WaitRead(0xc82024c060, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/fd_poll_runtime.go:78 +0x44
net.(*netFD).Read(0xc82024c000, 0xc820253000, 0x1000, 0x1000, 0x0, 0x5aa2050, 0xc8200121d0)
    /Users/tamird/src/go1.5/src/net/fd_unix.go:232 +0x27b
net.(*conn).Read(0xc820166070, 0xc820253000, 0x1000, 0x1000, 0x5aa6de0, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/net.go:172 +0x121
net.(*TCPConn).Read(0xc820166070, 0xc820253000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    <autogenerated>:74 +0x7d
bufio.(*Reader).fill(0xc82024e0c0)
    /Users/tamird/src/go1.5/src/bufio/bufio.go:97 +0x365
bufio.(*Reader).ReadByte(0xc82024e0c0, 0xc, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/bufio/bufio.go:229 +0x153
encoding/binary.ReadUvarint(0x5fa9d80, 0xc82024e0c0, 0x0, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/encoding/binary/varint.go:110 +0x60
github.com/cockroachdb/cockroach/rpc/codec.(*baseConn).recvProto(0xc820216180, 0x5fa9da8, 0xc8202162b8, 0x0, 0x4916ef8, 0x0, 0x0)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/codec/conn.go:88 +0x9e
github.com/cockroachdb/cockroach/rpc/codec.(*clientCodec).readResponseHeader(0xc820216180, 0xc8202162b8, 0x0, 0x0)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/codec/client.go:155 +0x8e
github.com/cockroachdb/cockroach/rpc/codec.(*clientCodec).ReadResponseHeader(0xc820216180, 0xc8201a9170, 0x0, 0x0)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/codec/client.go:80 +0x59
net/rpc.(*Client).input(0xc82024e120)
    /Users/tamird/src/go1.5/src/net/rpc/client.go:109 +0x177
created by net/rpc.NewClientWithCodec
    /Users/tamird/src/go1.5/src/net/rpc/client.go:201 +0x126

goroutine 70 [IO wait]:
net.runtime_pollWait(0x5fa5140, 0x72, 0x0)
    /Users/tamird/src/go1.5/src/runtime/netpoll.go:157 +0x63
net.(*pollDesc).Wait(0xc82024c140, 0x72, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/fd_poll_runtime.go:73 +0x56
net.(*pollDesc).WaitRead(0xc82024c140, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/fd_poll_runtime.go:78 +0x44
net.(*netFD).Read(0xc82024c0e0, 0xc820273000, 0x1000, 0x1000, 0x0, 0x5aa2050, 0xc8200121d0)
    /Users/tamird/src/go1.5/src/net/fd_unix.go:232 +0x27b
net.(*conn).Read(0xc820166080, 0xc820273000, 0x1000, 0x1000, 0xc8201b7280, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/net.go:172 +0x121
net.(*TCPConn).Read(0xc820166080, 0xc820273000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    <autogenerated>:74 +0x7d
bufio.(*Reader).fill(0xc82024e240)
    /Users/tamird/src/go1.5/src/bufio/bufio.go:97 +0x365
bufio.(*Reader).ReadByte(0xc82024e240, 0x403b1a5, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/bufio/bufio.go:229 +0x153
encoding/binary.ReadUvarint(0x5fa9d80, 0xc82024e240, 0x0, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/encoding/binary/varint.go:110 +0x60
github.com/cockroachdb/cockroach/rpc/codec.(*baseConn).recvProto(0xc820216780, 0x5fa9d48, 0xc8202168d8, 0x0, 0x4916ef8, 0x0, 0x0)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/codec/conn.go:88 +0x9e
github.com/cockroachdb/cockroach/rpc/codec.(*serverCodec).readRequestHeader(0xc820216780, 0xc82024e240, 0xc8202168d8, 0x0, 0x0)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/codec/server.go:173 +0x91
github.com/cockroachdb/cockroach/rpc/codec.(*serverCodec).ReadRequestHeader(0xc820216780, 0xc820566ce0, 0x0, 0x0)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/codec/server.go:60 +0x8c
github.com/cockroachdb/cockroach/rpc.(*Server).readRequest(0xc820122b00, 0x5fa9d08, 0xc820216780, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/server.go:403 +0xe7
github.com/cockroachdb/cockroach/rpc.(*Server).readRequests(0xc820122b00, 0x5fa9d08, 0xc820216780, 0xc82015c620, 0xc82024e2a0)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/server.go:361 +0xfa
github.com/cockroachdb/cockroach/rpc.(*Server).ServeHTTP(0xc820122b00, 0x5fa9c48, 0xc820268160, 0xc8202540e0)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/server.go:198 +0xace
net/http.serverHandler.ServeHTTP(0xc820132de0, 0x5fa9c48, 0xc820268160, 0xc8202540e0)
    /Users/tamird/src/go1.5/src/net/http/server.go:1862 +0x207
net/http.(*conn).serve(0xc8202680b0)
    /Users/tamird/src/go1.5/src/net/http/server.go:1361 +0x117d
created by net/http.(*Server).Serve
    /Users/tamird/src/go1.5/src/net/http/server.go:1910 +0x465

goroutine 71 [chan receive]:
github.com/cockroachdb/cockroach/rpc.(*Server).sendResponses(0xc820122b00, 0x5fa9d08, 0xc820216780, 0xc82024e2a0)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/server.go:426 +0xb5
github.com/cockroachdb/cockroach/rpc.(*Server).ServeHTTP.func1(0xc820122b00, 0x5fa9d08, 0xc820216780, 0xc82024e2a0, 0xc820162510)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/server.go:195 +0x4d
created by github.com/cockroachdb/cockroach/rpc.(*Server).ServeHTTP
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/server.go:197 +0xa8c

goroutine 92 [IO wait]:
net.runtime_pollWait(0x5fa5200, 0x72, 0x0)
    /Users/tamird/src/go1.5/src/runtime/netpoll.go:157 +0x63
net.(*pollDesc).Wait(0xc8201891e0, 0x72, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/fd_poll_runtime.go:73 +0x56
net.(*pollDesc).WaitRead(0xc8201891e0, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/fd_poll_runtime.go:78 +0x44
net.(*netFD).Read(0xc820189180, 0xc82027a000, 0x1000, 0x1000, 0x0, 0x5aa2050, 0xc8200121d0)
    /Users/tamird/src/go1.5/src/net/fd_unix.go:232 +0x27b
net.(*conn).Read(0xc8201202d8, 0xc82027a000, 0x1000, 0x1000, 0x5aa6de0, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/net.go:172 +0x121
net.(*TCPConn).Read(0xc8201202d8, 0xc82027a000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    <autogenerated>:74 +0x7d
bufio.(*Reader).fill(0xc820204480)
    /Users/tamird/src/go1.5/src/bufio/bufio.go:97 +0x365
bufio.(*Reader).ReadByte(0xc820204480, 0xc, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/bufio/bufio.go:229 +0x153
encoding/binary.ReadUvarint(0x5fa9d80, 0xc820204480, 0xc82003ebe8, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/encoding/binary/varint.go:110 +0x60
github.com/cockroachdb/cockroach/rpc/codec.(*baseConn).recvProto(0xc82027c000, 0x5fa9da8, 0xc82027c138, 0x0, 0x4916ef8, 0x0, 0x0)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/codec/conn.go:88 +0x9e
github.com/cockroachdb/cockroach/rpc/codec.(*clientCodec).readResponseHeader(0xc82027c000, 0xc82027c138, 0x0, 0x0)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/codec/client.go:155 +0x8e
github.com/cockroachdb/cockroach/rpc/codec.(*clientCodec).ReadResponseHeader(0xc82027c000, 0xc8200c7b90, 0x0, 0x0)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/codec/client.go:80 +0x59
net/rpc.(*Client).input(0xc8202044e0)
    /Users/tamird/src/go1.5/src/net/rpc/client.go:109 +0x177
created by net/rpc.NewClientWithCodec
    /Users/tamird/src/go1.5/src/net/rpc/client.go:201 +0x126

goroutine 72 [IO wait]:
net.runtime_pollWait(0x5fa5080, 0x72, 0x0)
    /Users/tamird/src/go1.5/src/runtime/netpoll.go:157 +0x63
net.(*pollDesc).Wait(0xc8202244c0, 0x72, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/fd_poll_runtime.go:73 +0x56
net.(*pollDesc).WaitRead(0xc8202244c0, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/fd_poll_runtime.go:78 +0x44
net.(*netFD).Read(0xc820224460, 0xc820277000, 0x1000, 0x1000, 0x0, 0x5aa2050, 0xc8200121d0)
    /Users/tamird/src/go1.5/src/net/fd_unix.go:232 +0x27b
net.(*conn).Read(0xc8201202f0, 0xc820277000, 0x1000, 0x1000, 0x5aa6de0, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/net.go:172 +0x121
net.(*TCPConn).Read(0xc8201202f0, 0xc820277000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    <autogenerated>:74 +0x7d
bufio.(*Reader).fill(0xc82024e300)
    /Users/tamird/src/go1.5/src/bufio/bufio.go:97 +0x365
bufio.(*Reader).ReadByte(0xc82024e300, 0xc, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/bufio/bufio.go:229 +0x153
encoding/binary.ReadUvarint(0x5fa9d80, 0xc82024e300, 0x0, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/encoding/binary/varint.go:110 +0x60
github.com/cockroachdb/cockroach/rpc/codec.(*baseConn).recvProto(0xc820216a80, 0x5fa9da8, 0xc820216bb8, 0x0, 0x4916ef8, 0x0, 0x0)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/codec/conn.go:88 +0x9e
github.com/cockroachdb/cockroach/rpc/codec.(*clientCodec).readResponseHeader(0xc820216a80, 0xc820216bb8, 0x0, 0x0)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/codec/client.go:155 +0x8e
github.com/cockroachdb/cockroach/rpc/codec.(*clientCodec).ReadResponseHeader(0xc820216a80, 0xc8200c7bc0, 0x0, 0x0)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/codec/client.go:80 +0x59
net/rpc.(*Client).input(0xc82024e360)
    /Users/tamird/src/go1.5/src/net/rpc/client.go:109 +0x177
created by net/rpc.NewClientWithCodec
    /Users/tamird/src/go1.5/src/net/rpc/client.go:201 +0x126

goroutine 23 [IO wait]:
net.runtime_pollWait(0x5fa4fc0, 0x72, 0x0)
    /Users/tamird/src/go1.5/src/runtime/netpoll.go:157 +0x63
net.(*pollDesc).Wait(0xc820224530, 0x72, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/fd_poll_runtime.go:73 +0x56
net.(*pollDesc).WaitRead(0xc820224530, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/fd_poll_runtime.go:78 +0x44
net.(*netFD).Read(0xc8202244d0, 0xc820284000, 0x1000, 0x1000, 0x0, 0x5aa2050, 0xc8200121d0)
    /Users/tamird/src/go1.5/src/net/fd_unix.go:232 +0x27b
net.(*conn).Read(0xc8200c2180, 0xc820284000, 0x1000, 0x1000, 0xc8201b9280, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/net.go:172 +0x121
net.(*TCPConn).Read(0xc8200c2180, 0xc820284000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    <autogenerated>:74 +0x7d
bufio.(*Reader).fill(0xc8200bc660)
    /Users/tamird/src/go1.5/src/bufio/bufio.go:97 +0x365
bufio.(*Reader).ReadByte(0xc8200bc660, 0x403b1a5, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/bufio/bufio.go:229 +0x153
encoding/binary.ReadUvarint(0x5fa9d80, 0xc8200bc660, 0x0, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/encoding/binary/varint.go:110 +0x60
github.com/cockroachdb/cockroach/rpc/codec.(*baseConn).recvProto(0xc820065200, 0x5fa9d48, 0xc820065358, 0x0, 0x4916ef8, 0x0, 0x0)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/codec/conn.go:88 +0x9e
github.com/cockroachdb/cockroach/rpc/codec.(*serverCodec).readRequestHeader(0xc820065200, 0xc8200bc660, 0xc820065358, 0x0, 0x0)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/codec/server.go:173 +0x91
github.com/cockroachdb/cockroach/rpc/codec.(*serverCodec).ReadRequestHeader(0xc820065200, 0xc820566c60, 0x0, 0x0)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/codec/server.go:60 +0x8c
github.com/cockroachdb/cockroach/rpc.(*Server).readRequest(0xc820122c00, 0x5fa9d08, 0xc820065200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/server.go:403 +0xe7
github.com/cockroachdb/cockroach/rpc.(*Server).readRequests(0xc820122c00, 0x5fa9d08, 0xc820065200, 0xc8200bedc0, 0xc8200bc6c0)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/server.go:361 +0xfa
github.com/cockroachdb/cockroach/rpc.(*Server).ServeHTTP(0xc820122c00, 0x5fa9c48, 0xc820242210, 0xc82014e2a0)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/server.go:198 +0xace
net/http.serverHandler.ServeHTTP(0xc8201ce180, 0x5fa9c48, 0xc820242210, 0xc82014e2a0)
    /Users/tamird/src/go1.5/src/net/http/server.go:1862 +0x207
net/http.(*conn).serve(0xc820242160)
    /Users/tamird/src/go1.5/src/net/http/server.go:1361 +0x117d
created by net/http.(*Server).Serve
    /Users/tamird/src/go1.5/src/net/http/server.go:1910 +0x465

goroutine 24 [chan receive]:
github.com/cockroachdb/cockroach/rpc.(*Server).sendResponses(0xc820122c00, 0x5fa9d08, 0xc820065200, 0xc8200bc6c0)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/server.go:426 +0xb5
github.com/cockroachdb/cockroach/rpc.(*Server).ServeHTTP.func1(0xc820122c00, 0x5fa9d08, 0xc820065200, 0xc8200bc6c0, 0xc82013b410)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/server.go:195 +0x4d
created by github.com/cockroachdb/cockroach/rpc.(*Server).ServeHTTP
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/server.go:197 +0xa8c

goroutine 131 [IO wait]:
net.runtime_pollWait(0x5fa4f00, 0x72, 0x0)
    /Users/tamird/src/go1.5/src/runtime/netpoll.go:157 +0x63
net.(*pollDesc).Wait(0xc8202245a0, 0x72, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/fd_poll_runtime.go:73 +0x56
net.(*pollDesc).WaitRead(0xc8202245a0, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/fd_poll_runtime.go:78 +0x44
net.(*netFD).Read(0xc820224540, 0xc8202bf000, 0x1000, 0x1000, 0x0, 0x5aa2050, 0xc8200121d0)
    /Users/tamird/src/go1.5/src/net/fd_unix.go:232 +0x27b
net.(*conn).Read(0xc8200c2288, 0xc8202bf000, 0x1000, 0x1000, 0x5aa6de0, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/net.go:172 +0x121
net.(*TCPConn).Read(0xc8200c2288, 0xc8202bf000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    <autogenerated>:74 +0x7d
bufio.(*Reader).fill(0xc8200bcc60)
    /Users/tamird/src/go1.5/src/bufio/bufio.go:97 +0x365
bufio.(*Reader).ReadByte(0xc8200bcc60, 0xc, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/bufio/bufio.go:229 +0x153
encoding/binary.ReadUvarint(0x5fa9d80, 0xc8200bcc60, 0x0, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/encoding/binary/varint.go:110 +0x60
github.com/cockroachdb/cockroach/rpc/codec.(*baseConn).recvProto(0xc820065800, 0x5fa9da8, 0xc820065938, 0x0, 0x4916ef8, 0x0, 0x0)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/codec/conn.go:88 +0x9e
github.com/cockroachdb/cockroach/rpc/codec.(*clientCodec).readResponseHeader(0xc820065800, 0xc820065938, 0x0, 0x0)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/codec/client.go:155 +0x8e
github.com/cockroachdb/cockroach/rpc/codec.(*clientCodec).ReadResponseHeader(0xc820065800, 0xc820161b00, 0x0, 0x0)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/codec/client.go:80 +0x59
net/rpc.(*Client).input(0xc8200bccc0)
    /Users/tamird/src/go1.5/src/net/rpc/client.go:109 +0x177
created by net/rpc.NewClientWithCodec
    /Users/tamird/src/go1.5/src/net/rpc/client.go:201 +0x126

goroutine 105 [chan receive]:
github.com/cockroachdb/cockroach/rpc.(*Server).sendResponses(0xc820122c80, 0x5fa9d08, 0xc82027c780, 0xc820204de0)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/server.go:426 +0xb5
github.com/cockroachdb/cockroach/rpc.(*Server).ServeHTTP.func1(0xc820122c80, 0x5fa9d08, 0xc82027c780, 0xc820204de0, 0xc820250570)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/server.go:195 +0x4d
created by github.com/cockroachdb/cockroach/rpc.(*Server).ServeHTTP
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/server.go:197 +0xa8c

goroutine 151 [IO wait]:
net.runtime_pollWait(0x5fa4e40, 0x72, 0x0)
    /Users/tamird/src/go1.5/src/runtime/netpoll.go:157 +0x63
net.(*pollDesc).Wait(0xc82024c1b0, 0x72, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/fd_poll_runtime.go:73 +0x56
net.(*pollDesc).WaitRead(0xc82024c1b0, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/fd_poll_runtime.go:78 +0x44
net.(*netFD).Read(0xc82024c150, 0xc82027f000, 0x1000, 0x1000, 0x0, 0x5aa2050, 0xc8200121d0)
    /Users/tamird/src/go1.5/src/net/fd_unix.go:232 +0x27b
net.(*conn).Read(0xc8201661b8, 0xc82027f000, 0x1000, 0x1000, 0xc82004b280, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/net/net.go:172 +0x121
net.(*TCPConn).Read(0xc8201661b8, 0xc82027f000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    <autogenerated>:74 +0x7d
bufio.(*Reader).fill(0xc820204d80)
    /Users/tamird/src/go1.5/src/bufio/bufio.go:97 +0x365
bufio.(*Reader).ReadByte(0xc820204d80, 0x403b1a5, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/bufio/bufio.go:229 +0x153
encoding/binary.ReadUvarint(0x5fa9d80, 0xc820204d80, 0x0, 0x0, 0x0)
    /Users/tamird/src/go1.5/src/encoding/binary/varint.go:110 +0x60
github.com/cockroachdb/cockroach/rpc/codec.(*baseConn).recvProto(0xc82027c780, 0x5fa9d48, 0xc82027c8d8, 0x0, 0x4916ef8, 0x0, 0x0)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/codec/conn.go:88 +0x9e
github.com/cockroachdb/cockroach/rpc/codec.(*serverCodec).readRequestHeader(0xc82027c780, 0xc820204d80, 0xc82027c8d8, 0x0, 0x0)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/codec/server.go:173 +0x91
github.com/cockroachdb/cockroach/rpc/codec.(*serverCodec).ReadRequestHeader(0xc82027c780, 0xc820566cc0, 0x0, 0x0)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/codec/server.go:60 +0x8c
github.com/cockroachdb/cockroach/rpc.(*Server).readRequest(0xc820122c80, 0x5fa9d08, 0xc82027c780, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/server.go:403 +0xe7
github.com/cockroachdb/cockroach/rpc.(*Server).readRequests(0xc820122c80, 0x5fa9d08, 0xc82027c780, 0xc82011fca0, 0xc820204de0)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/server.go:361 +0xfa
github.com/cockroachdb/cockroach/rpc.(*Server).ServeHTTP(0xc820122c80, 0x5fa9c48, 0xc820128840, 0xc820130c40)
    /Users/tamird/src/go/src/github.com/cockroachdb/cockroach/rpc/server.go:198 +0xace
net/http.serverHandler.ServeHTTP(0xc8201ceb40, 0x5fa9c48, 0xc820128840, 0xc820130c40)
    /Users/tamird/src/go1.5/src/net/http/server.go:1862 +0x207
net/http.(*conn).serve(0xc8202682c0)
    /Users/tamird/src/go1.5/src/net/http/server.go:1361 +0x117d
created by net/http.(*Server).Serve
    /Users/tamird/src/go1.5/src/net/http/server.go:1910 +0x465

real    3m12.755s
user    2m42.945s
sys 0m39.444s

@tamird tamird reopened this Nov 5, 2015
@tamird tamird changed the title TestRemoveLeader deadlocks in multi cpu multiraft: TestRemoveLeader deadlocks in multi cpu Nov 5, 2015
@tamird tamird mentioned this issue Nov 12, 2015
Merged
@tamird tamird removed the multi-cpu label Nov 13, 2015
bdarnell added a commit to bdarnell/cockroach that referenced this issue Nov 17, 2015
@bdarnell
raft: Remove waitForCallback and centralize re-proposal logic.
285a7fa 
When a proposal is forwarded from a follower to a leader,
waitForCallback is incorrect: what matters is whether the *leader* is
waiting for a pending config change callback. Instead, we need to
repropose from the follower when the leader has completed its pending
config change. This is difficult to observe directly but does have a
side effect: when a config change is dropped, it is replaced with an
empty entry. When we see such an entry, we know we have to re-propose.

Since empty entries are also used to signal leader changes, we can
remove the re-proposals in maybeSendLeaderEvent.

Fixes cockroachdb#2639
@bdarnell bdarnell mentioned this issue Nov 17, 2015
Merged
bdarnell added a commit to bdarnell/cockroach that referenced this issue Nov 17, 2015
@bdarnell
raft: Remove waitForCallback and centralize re-proposal logic.
c871371 
When a proposal is forwarded from a follower to a leader,
waitForCallback is incorrect: what matters is whether the *leader* is
waiting for a pending config change callback. Instead, we need to
repropose from the follower when the leader has completed its pending
config change. This is difficult to observe directly but does have a
side effect: when a config change is dropped, it is replaced with an
empty entry. When we see such an entry, we know we have to re-propose.

Since empty entries are also used to signal leader changes, we can
remove the re-proposals in maybeSendLeaderEvent.

Fixes cockroachdb#2639
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bdarnell bdarnell

Labels
C-test-failure Broken test (automatically or manually discovered).
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@bdarnell @tamird