Self-monitoring panic after inserting data

[mark-rushakoff]

Built from source at 615aabf.

After starting InfluxDB with an empty data directory, running this command:

curl -G http://localhost:8086/query --data-urlencode "q=CREATE DATABASE foo" && \
  curl -d '{"database":"foo", "points": [{"name": "cpu","fields": {"value": 100}}]}' -H "Content-Type: application/json" http://localhost:8086/write

causes the server to panic:

[messaging] 2015/04/24 14:17:14 connected to broker: http://:8086/messaging/messages?index=0&streaming=true&topicID=1
[server] 2015/04/24 14:17:14 retention policy 'default' auto-created for database 'foo'
[http] 2015/04/24 14:17:14 ::1 - - [24/Apr/2015:14:17:14 -0700] GET /query?q=CREATE%20DATABASE%20foo HTTP/1.1 200 16 - curl/7.37.1 476a712d-eac7-11e4-800b-000000000000 64.122012ms
[http] 2015/04/24 14:17:14 ::1 - - [24/Apr/2015:14:17:14 -0700] POST /write HTTP/1.1 200 0 - curl/7.37.1 4775dbf6-eac7-11e4-800c-000000000000 191.355976ms
[messaging] 2015/04/24 14:17:14 connected to broker: http://:8086/messaging/messages?index=0&streaming=true&topicID=2
panic: runtime error: index out of range

goroutine 259 [running]:
github.com/influxdb/influxdb.(*Server).convertRowToPoints(0xc20805ab00, 0x5e4b70, 0xb, 0xc2081233e0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/mr/go/src/github.com/influxdb/influxdb/server.go:4018 +0x7b7
github.com/influxdb/influxdb.func·011()
    /Users/mr/go/src/github.com/influxdb/influxdb/server.go:399 +0x5c0
created by github.com/influxdb/influxdb.(*Server).StartSelfMonitoring
    /Users/mr/go/src/github.com/influxdb/influxdb/server.go:412 +0x241

[otoolep]

Thanks @mark-rushakoff -- I'll take a look.

[otoolep]

Cannot reproduce this. Blew away the data, and started a single brand-new node.

$ curl -v -G http://localhost:8086/query --data-urlencode "q=CREATE DATABASE foo"
* About to connect() to localhost port 8086 (#0)
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 8086 (#0)
> GET /query?q=CREATE%20DATABASE%20foo HTTP/1.1
> User-Agent: curl/7.29.0
> Host: localhost:8086
> Accept: */*
> 
< HTTP/1.1 200 OK
< Content-Type: application/json
< Request-Id: fa6fa00d-eae3-11e4-8007-000000000000
< X-Influxdb-Version: 0.9
< Date: Sat, 25 Apr 2015 00:42:40 GMT
< Content-Length: 16
< 
* Connection #0 to host localhost left intact
$ curl -v -d '{"database":"foo", "points": [{"name": "cpu","fields": {"value": 100}}]}' -H "Content-Type: application/json" http://localhost:8086/write
* About to connect() to localhost port 8086 (#0)
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 8086 (#0)
> POST /write HTTP/1.1
> User-Agent: curl/7.29.0
> Host: localhost:8086
> Accept: */*
> Content-Type: application/json
> Content-Length: 72
> 
* upload completely sent off: 72 out of 72 bytes
< HTTP/1.1 200 OK
< Request-Id: ff0c02cf-eae3-11e4-8010-000000000000
< X-Influxdb-Version: 0.9
< Date: Sat, 25 Apr 2015 00:42:48 GMT
< Content-Length: 0
< Content-Type: text/plain; charset=utf-8
< 
* Connection #0 to host localhost left intact

[marcosnils]

@otoolep this is the issue I was having here: #2422 (comment). In my case starting a fresh node (deleting the data dir) fixed the problem but seems like there might be something in there.

[marcosnils]

@otoolep the problem just appeared again after restarting the server a couple of times. Seems like the shards_diag self monitoring metrics are not pre-processed correctly and that raises the panic error.

This is how I'm seeing the shards_diag row get's processed:

[2015-04-25 01:15:03.956810841 +0000 UTC 1 1 550 /home/marcos/.influxdb/data/shards/1 /home/marcos/.influxdb/data/shards/2] map[id:1 dataNodes:2 index:3 path:5]
[2015-04-25 01:15:03.956810841 +0000 UTC 2 1 221] map[index:3 path:5 id:1 dataNodes:2]

As you can see in the second line the "path" column is there even though there's no value for that. I don't know how shards are being handled internally but I believe that the problem must be around there.

[otoolep]

@marcosnils -- could you please open a separate ticket for this issue, and include the details above (and anything else that will help me repro this).

[marcosnils]

@otoolep found the error in the code I'll send a PR shortly. Seems like an easy fix

[otoolep]

@mark-rushakoff -- I am pretty sure the issue is actually the same after all, as @marcosnils was pointing out. Should be fixed by #2431

[mark-rushakoff]

Confirmed fixed as of 86b9ec5.