[0.9.4.2] Intermittent OpenTSDB plugin panic

[gunnaraasen]

Every once in awhile, when the OpenTSDB plugin is enabled using the b1 engine, the process will fall into a loop where the plugin panics and is recovered every few seconds. The behavior seems to occur periodically after every few restarts. Also, this is with Monit running as the process manager.

OpenTSDB config section:

[opentsdb]
  enabled = true
  bind-address = ":4242"
  database = "opentsdb"
  retention-policy = ""
  consistency-level = "one"
  tls-enabled = true
  certificate = "/etc/ssl/influxdb.pem"

Stack trace:

[opentsdb] 2015/10/13 18:13:30 Starting OpenTSDB service
[opentsdb] 2015/10/13 18:13:30 Listening on TLS: [::]:4242
[retention] 2015/10/13 18:13:30 Starting retention policy enforcement service with check interval of 10m0s
[run] 2015/10/13 18:13:30 Listening for signals
2015/10/13 18:13:30 Sending anonymous usage statistics to m.influxdb.com
[http] 2015/10/13 18:13:51 55.158.17.175 - - [13/Oct/2015:18:13:51 -0400] GET /ping HTTP/1.1 204 0 - Go-http-client/1.1 af03a4a4-71f7-11e5-8001-000000000000 73.676µs
2015/10/13 18:13:55 InfluxDB starting, version 0.9.4.2, branch 0.9.4, commit da1b59e7d7764d36786253b2db13b97f42ed4e1d
2015/10/13 18:13:55 Go version go1.5, GOMAXPROCS set to 4
2015/10/13 18:13:55 Using configuration at: /etc/opt/influxdb/influxdb.conf
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x50 pc=0x5d0588]

goroutine 1 [running]:
github.com/influxdb/influxdb/tsdb.(*PointBatcher).Stop(0x0)
    /tmp/build/src/github.com/influxdb/influxdb/tsdb/batcher.go:112 +0x18
github.com/influxdb/influxdb/services/opentsdb.(*Service).Close(0xc820182000, 0x0, 0x0)
    /tmp/build/src/github.com/influxdb/influxdb/services/opentsdb/service.go:172 +0x75
github.com/influxdb/influxdb/cmd/influxd/run.(*Server).Close(0xc82014b8c0, 0x0, 0x0)
    /tmp/build/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:404 +0xdf
github.com/influxdb/influxdb/cmd/influxd/run.(*Server).Open(0xc82014b8c0, 0x0, 0x0)
    /tmp/build/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:385 +0x55
github.com/influxdb/influxdb/cmd/influxd/run.(*Command).Run(0xc820134000, 0xc82000a060, 0x4, 0x4, 0x0, 0x0)
    /tmp/build/src/github.com/influxdb/influxdb/cmd/influxd/run/command.go:114 +0xd7e
main.(*Main).Run(0xc82010bf00, 0xc82000a060, 0x4, 0x4, 0x0, 0x0)
    /tmp/build/src/github.com/influxdb/influxdb/cmd/influxd/main.go:81 +0x67e
main.main()
    /tmp/build/src/github.com/influxdb/influxdb/cmd/influxd/main.go:42 +0x29b

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /home/philip/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1

goroutine 5 [syscall]:
os/signal.loop()
    /home/philip/.gvm/gos/go1.5/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
    /home/philip/.gvm/gos/go1.5/src/os/signal/signal_unix.go:28 +0x37
2015/10/13 18:14:40 InfluxDB starting, version 0.9.4.2, branch 0.9.4, commit da1b59e7d7764d36786253b2db13b97f42ed4e1d
2015/10/13 18:14:40 Go version go1.5, GOMAXPROCS set to 4
2015/10/13 18:14:40 Using configuration at: /etc/opt/influxdb/influxdb.conf
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x50 pc=0x5d0588]

goroutine 1 [running]:
github.com/influxdb/influxdb/tsdb.(*PointBatcher).Stop(0x0)
    /tmp/build/src/github.com/influxdb/influxdb/tsdb/batcher.go:112 +0x18
github.com/influxdb/influxdb/services/opentsdb.(*Service).Close(0xc8200b2620, 0x0, 0x0)
    /tmp/build/src/github.com/influxdb/influxdb/services/opentsdb/service.go:172 +0x75
github.com/influxdb/influxdb/cmd/influxd/run.(*Server).Close(0xc8201270e0, 0x0, 0x0)
    /tmp/build/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:404 +0xdf
github.com/influxdb/influxdb/cmd/influxd/run.(*Server).Open(0xc8201270e0, 0x0, 0x0)
    /tmp/build/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:385 +0x55
github.com/influxdb/influxdb/cmd/influxd/run.(*Command).Run(0xc8200ac380, 0xc82007a010, 0x4, 0x4, 0x0, 0x0)
    /tmp/build/src/github.com/influxdb/influxdb/cmd/influxd/run/command.go:114 +0xd7e
main.(*Main).Run(0xc82010df00, 0xc82007a010, 0x4, 0x4, 0x0, 0x0)
    /tmp/build/src/github.com/influxdb/influxdb/cmd/influxd/main.go:81 +0x67e
main.main()
    /tmp/build/src/github.com/influxdb/influxdb/cmd/influxd/main.go:42 +0x29b

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /home/philip/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1

goroutine 19 [syscall]:
os/signal.loop()
    /home/philip/.gvm/gos/go1.5/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
    /home/philip/.gvm/gos/go1.5/src/os/signal/signal_unix.go:28 +0x37
[http] 2015/10/13 18:14:50 55.158.17.175 - - [13/Oct/2015:18:14:50 -0400] GET /ping HTTP/1.1 204 0 - Go-http-client/1.1 d2ab9819-71f7-11e5-8002-000000000000 44.602µs
2015/10/13 18:15:25 InfluxDB starting, version 0.9.4.2, branch 0.9.4, commit da1b59e7d7764d36786253b2db13b97f42ed4e1d
2015/10/13 18:15:25 Go version go1.5, GOMAXPROCS set to 4
2015/10/13 18:15:26 Using configuration at: /etc/opt/influxdb/influxdb.conf
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x50 pc=0x5d0588]

goroutine 1 [running]:
github.com/influxdb/influxdb/tsdb.(*PointBatcher).Stop(0x0)
    /tmp/build/src/github.com/influxdb/influxdb/tsdb/batcher.go:112 +0x18
github.com/influxdb/influxdb/services/opentsdb.(*Service).Close(0xc8200b2620, 0x0, 0x0)
    /tmp/build/src/github.com/influxdb/influxdb/services/opentsdb/service.go:172 +0x75
github.com/influxdb/influxdb/cmd/influxd/run.(*Server).Close(0xc820121440, 0x0, 0x0)
    /tmp/build/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:404 +0xdf
github.com/influxdb/influxdb/cmd/influxd/run.(*Server).Open(0xc820121440, 0x0, 0x0)
    /tmp/build/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:385 +0x55
github.com/influxdb/influxdb/cmd/influxd/run.(*Command).Run(0xc8200ac380, 0xc82007a010, 0x4, 0x4, 0x0, 0x0)
    /tmp/build/src/github.com/influxdb/influxdb/cmd/influxd/run/command.go:114 +0xd7e
main.(*Main).Run(0xc82010df00, 0xc82007a010, 0x4, 0x4, 0x0, 0x0)
    /tmp/build/src/github.com/influxdb/influxdb/cmd/influxd/main.go:81 +0x67e
main.main()
    /tmp/build/src/github.com/influxdb/influxdb/cmd/influxd/main.go:42 +0x29b

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /home/philip/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1

goroutine 20 [syscall]:
os/signal.loop()
    /home/philip/.gvm/gos/go1.5/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
    /home/philip/.gvm/gos/go1.5/src/os/signal/signal_unix.go:28 +0x37

[otoolep]

What seems to have happened here is that the influxd process was shut down almost immediately after it was started, which exposed a bug in the Close() call.

[otoolep]

Fixed by #4438