panic on kapacitor restart

[rubycut]

Kapacitor 1.0.0

Server couldn't be restarted, because it was constantly crashing while loading tasks on startup:

Dec 12 13:48:13  kapacitord[323]: [c3605_sbm:min3] 2016/12/12 13:48:13 E! failed to aggregate batch: field Seconds_Behind_Master missing from
Dec 12 13:48:13 kapacitord[323]: panic: runtime error: invalid memory address or nil pointer dereference
Dec 12 13:48:13 kapacitord[323]: [signal 0xb code=0x1 addr=0x0 pc=0x609ea5]
Dec 12 13:48:13 kapacitord[323]: goroutine 11824 [running]:
Dec 12 13:48:13 kapacitord[323]: panic(0xd68920, 0xc820010080)
Dec 12 13:48:13 kapacitord[323]: /usr/local/go/src/runtime/panic.go:481 +0x3e6
Dec 12 13:48:13 kapacitord[323]: github.com/influxdata/kapacitor/vendor/github.com/influxdata/influxdb/influxql.(*FloatFuncReducer).Emit(0xc
Dec 12 13:48:13 kapacitord[323]: /root/go/src/github.com/influxdata/kapacitor/vendor/github.com/influxdata/influxdb/influxql/functions.gen.g
Dec 12 13:48:13 kapacitord[323]: github.com/influxdata/kapacitor.(*floatPointEmitter).EmitPoint(0xc856acb290, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
Dec 12 13:48:13 kapacitord[323]: /root/go/src/github.com/influxdata/kapacitor/influxql.gen.go:167 +0x7c
Dec 12 13:48:13 kapacitord[323]: github.com/influxdata/kapacitor.(*InfluxQLNode).emit(0xc84c477500, 0x7f53e404d050, 0xc856acb260, 0x0, 0x0)
Dec 12 13:48:13 kapacitord[323]: /root/go/src/github.com/influxdata/kapacitor/influxql.go:231 +0x90

[rubycut]

Template was added first:

kapacitor define c3605_sbm -template sbm_monitor -vars /tmp/kapacitor/c3605.json  -dbrp c3605.four_months

JSON:

# cat /tmp/kapacitor/c3605.json
{ "customer": { "type":"string", "value":"c3605" } }

Template itself:

# cat sbm_template.tick
// ver 0.1
// 09.12.2016

var customer string


var data = stream
    |from()
        .database(customer)
        .retentionPolicy('four_months')
        .measurement('xxx_mysql_slave')
        .groupBy('host')
    |window()
        .period(300s)
        .every(60s)

var average = data
    |min('Seconds_Behind_Master')
    |alert()
        .crit(lambda: "min" > 1000 )
        .message('MySQL slave: Seconds Behind Master {{ .Level }} >  1000s on {{ index .Tags "host" }}: {{ index .Fields "min" }} s')
        .email()
        .to('email1@example.com')
        .to('email2@example.com')
        .stateChangesOnly()

[nathanielc]

@rubycut Thanks for the detailed report. I'll take a look at this. If it helps you get up and running, that stacktrace is for a path that is processing data. Meaning the panic is coming from writing data to Kapacitor. If you disable all writes for a bit you should be able to start up Kapacitor and disable the offending task.

Either way we will get a fix for this.

[rashtisouciance]

I am seeing something similar when starting Kapacitor 1.1.1 from docker.

2017/01/23 14:17:05 Using configuration at: /etc/kapacitor/kapacitor.conf
[run] 2017/01/23 14:17:05 I! Kapacitor starting, version 1.1.1, branch master, commit d1e0173
[run] 2017/01/23 14:17:05 I! Go version go1.7.4
[srv] 2017/01/23 14:17:05 I! Kapacitor hostname: 5088cb5b916e
[srv] 2017/01/23 14:17:05 I! ClusterID: f95e8c23-01c3-4337-9fdf-c83eb064bda4 ServerID: 9ca4a62c-1ce6-4ac7-bbf7-1675af8d9249
[task_master:main] 2017/01/23 14:17:05 I! opened
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x54f153]

goroutine 1 [running]:
panic(0xb08ee0, 0xc42000c060)
        /usr/local/go/src/runtime/panic.go:500 +0x1a1
github.com/influxdata/kapacitor/services/influxdb.(*influxdbCluster).Close(0xc4200ccd20, 0x0, 0x0)
        /root/go/src/github.com/influxdata/kapacitor/services/influxdb/service.go:518 +0xa3
github.com/influxdata/kapacitor/services/influxdb.(*Service).Close(0xc420180b00, 0x0, 0x0)
        /root/go/src/github.com/influxdata/kapacitor/services/influxdb/service.go:143 +0x145
github.com/influxdata/kapacitor/server.(*Server).Close(0xc42007db80, 0xff82e0, 0xc4201e3d30)
        /root/go/src/github.com/influxdata/kapacitor/server/server.go:626 +0x14a
github.com/influxdata/kapacitor/server.(*Server).Open(0xc42007db80, 0xc5209c, 0x5)
        /root/go/src/github.com/influxdata/kapacitor/server/server.go:555 +0x94
github.com/influxdata/kapacitor/cmd/kapacitord/run.(*Command).Run(0xc420080900, 0xc42000c220, 0x0, 0x0, 0xc420091828, 0x108)
        /root/go/src/github.com/influxdata/kapacitor/cmd/kapacitord/run/command.go:127 +0xb10
main.(*Main).Run(0xc4201a5ef0, 0xc42000c220, 0x0, 0x0, 0xc42004bf38, 0x40740b)
        /root/go/src/github.com/influxdata/kapacitor/cmd/kapacitord/main.go:80 +0x2e2
main.main()
        /root/go/src/github.com/influxdata/kapacitor/cmd/kapacitord/main.go:41 +0x20b
[root@ip-172-31-35-12 kapacitor]#

I don't have any tasks as I just wanted to see Kapacitor starting. What could be the issue here? I am downloading image 1.1.1.

[nathanielc]

@rubycut The panic from your stack trace was fixed in #1126

@rashtisouciance The panic you reported will be fixed in #1141

Thanks

[rashtisouciance]

@nathanielc , great thanks.

[randomeleven]

I am sorry to reopen this issue.

But I am having a similar problem when trying to start kapacitor, start/restart commands fail.

• Ubuntu 14.04 LTS
• Kapacitor 1.2.0 (git: master 5408057)
• Telegraf v1.2.1 (git: release-1.2 3b6ffb344e5c03c1595d862282a6823ecb438cff)
• Influxdb version: 1.2.0

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x51c319]

goroutine 1 [running]:
panic(0xb6fa40, 0xc4200100a0)
        /usr/local/go/src/runtime/panic.go:500 +0x1a1
github.com/influxdata/kapacitor/server.(*Server).setupIDs(0xc4201ba180, 0xc4201a                                                                                                                     d858, 0x2)
        /root/go/src/github.com/influxdata/kapacitor/server/server.go:715 +0x79
github.com/influxdata/kapacitor/server.New(0xc42013e080, 0xccd314, 0x5, 0xcd31c0                                                                                                                     , 0x28, 0xccd31c, 0x6, 0x10ae1a0, 0xc42000d7c0, 0x0, ...)
        /root/go/src/github.com/influxdata/kapacitor/server/server.go:142 +0x55f
github.com/influxdata/kapacitor/cmd/kapacitord/run.(*Command).Run(0xc420082a20,                                                                                                                      0xc42000c0b0, 0x4, 0x4, 0xc420091801, 0x108)
        /root/go/src/github.com/influxdata/kapacitor/cmd/kapacitord/run/command.                                                                                                                     go:121 +0xa81
main.(*Main).Run(0xc4201adef0, 0xc42000c0b0, 0x4, 0x4, 0xc42004bf38, 0x40740b)
        /root/go/src/github.com/influxdata/kapacitor/cmd/kapacitord/main.go:80 +                                                                                                                     0x2e2
main.main()
        /root/go/src/github.com/influxdata/kapacitor/cmd/kapacitord/main.go:41 +                                                                                                                     0x20b