panic: runtime error: invalid memory address or nil pointer dereference (maybe when another cluster node is removed)

[jpetazzo]

I just experienced the following crash of my engine.

It looks like this happens has I forcibly removed another node from the cluster (essentially I powercycled node3 and noticed that I had lost my Swarm leader, even though node3 was not leader at that time; then I noticed that I had lost node2: the Engine had crashed).

Mar 02 20:44:17 node2 docker[2728]: panic: runtime error: invalid memory address or nil pointer dereference
Mar 02 20:44:17 node2 docker[2728]: [signal 0xb code=0x1 addr=0x0 pc=0xb6c858]
Mar 02 20:44:17 node2 docker[2728]: goroutine 976 [running]:
Mar 02 20:44:17 node2 docker[2728]: github.com/docker/libnetwork/drivers/overlay.(*network).endpoint(0x0, 0xc8202fa1c0, 0x40, 0x0)
Mar 02 20:44:17 node2 docker[2728]: /usr/src/docker/vendor/src/github.com/docker/libnetwork/drivers/overlay/ov_endpoint.go:23 +0xe8
Mar 02 20:44:17 node2 docker[2728]: github.com/docker/libnetwork/drivers/overlay.(*driver).notifyEvent(0xc820081860, 0x1655770, 0x4, 0xc8202fa1c0, 0x40, 0xc821464540, 0x40)
Mar 02 20:44:17 node2 docker[2728]: /usr/src/docker/vendor/src/github.com/docker/libnetwork/drivers/overlay/ov_serf.go:85 +0x77
Mar 02 20:44:17 node2 docker[2728]: github.com/docker/libnetwork/drivers/overlay.(*driver).startSerfLoop(0xc820081860, 0xc8211ad800, 0xc8211adc80, 0xc8211adce0)
Mar 02 20:44:17 node2 docker[2728]: /usr/src/docker/vendor/src/github.com/docker/libnetwork/drivers/overlay/ov_serf.go:194 +0x196
Mar 02 20:44:17 node2 docker[2728]: created by github.com/docker/libnetwork/drivers/overlay.(*driver).serfInit
Mar 02 20:44:17 node2 docker[2728]: /usr/src/docker/vendor/src/github.com/docker/libnetwork/drivers/overlay/ov_serf.go:68 +0x496

Full log is attached. I'll add details if it happens again (I'm doing a lot of chaos-monkey style failures so maybe it'll happen again...)

Server:
 Version:      1.10.2
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   c3959b1
 Built:        Mon Feb 22 21:40:35 2016
 OS/Arch:      linux/amd64

docker.log.txt

[pdevine]

I hit something really similar while attempting to set up host discovery with UCP/Swarm. I was doing some crazy stuff to try and diagnose a problem with host discovery, so I'm not exactly sure what triggered it. It could be that I updated the clock, upgraded the engine or even messed around with the TLS certs.

`panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x38 pc=0xb71a41]

goroutine 1548 [running]:
github.com/docker/libnetwork/drivers/overlay.(_driver).notifyEvent(0xc82068bf40, 0x1648d30, 0x4, 0xc8214fcd00, 0x40, 0xc82107a740, 0x40)
/usr/src/docker/vendor/src/github.com/docker/libnetwork/drivers/overlay/ov_serf.go:87 +0x81
github.com/docker/libnetwork/drivers/overlay.(_driver).startSerfLoop(0xc82068bf40, 0xc82084d740, 0xc820bd0180, 0xc820bd01e0)
/usr/src/docker/vendor/src/github.com/docker/libnetwork/drivers/overlay/ov_serf.go:194 +0x196
created by github.com/docker/libnetwork/drivers/overlay.(*driver).serfInit
/usr/src/docker/vendor/src/github.com/docker/libnetwork/drivers/overlay/ov_serf.go:68 +0x496

goroutine 1 [chan receive, 81 minutes]:
main.(_DaemonCli).CmdDaemon(0xc82030fa40, 0xc82000a090, 0x0, 0x0, 0x0, 0x0)
/usr/src/docker/docker/daemon.go:305 +0x20a7
reflect.callMethod(0xc820644de0, 0xc820705c78)
/usr/local/go/src/reflect/value.go:628 +0x1fc
reflect.methodValueCall(0xc82000a090, 0x0, 0x0, 0x1, 0xc820644de0, 0x0, 0x0, 0xc820644de0, 0xc820646990, 0x47a034, ...)
/usr/local/go/src/reflect/asm_amd64.s:29 +0x36
github.com/docker/docker/cli.(_Cli).Run(0xc820644d50, 0xc82000a090, 0x1, 0x1, 0x0, 0x0)
/usr/src/docker/.gopath/src/github.com/docker/docker/cli/cli.go:89 +0x383
main.main()
/usr/src/docker/docker/docker.go:63 +0x43c

goroutine 17 [syscall, 84 minutes, locked to thread]:
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1721 +0x1
`

[mavenugo]

Thanks @jpetazzo @pdevine. We have a proposed fix in #1050. We will look into the root-cause and get this resolved for 1.11. thanks.