[0.13.0] Can't start database with too many series

[josselin-c]

Bug report

System info:
On debian Jessie - 32GB of ram dedicated to influxdb

InfluxDB starting, version 0.13.0, branch 0.13, commit e57fb88
Go version go1.6.2, GOMAXPROCS set to 8

Steps to reproduce:

Expected behavior:
Influxdb should restart correctly even when killed improperly
Actual behavior:
After an OOM Killer, influxdb isn't able to restart again and keeps crashing
Removing the folder containing the corrupted? database, influxdb restarts correctly

Additional info:
Here is the stacktrace:

[cacheloader] 2016/06/23 13:46:17 reading file /home/influxdb/wal/mydb/default/235/_00020.wal, size 10485802

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x886334]
goroutine 280 [running]:
panic(0xb5b240, 0xc82000e0c0)
        /usr/local/go/src/runtime/panic.go:481 +0x3e6
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*entry).size(0x0, 0xc83e643980)
        /root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/cache.go:101 +0x94
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Cache).DeleteRange(0xc842343e80, 0xc84ee94000, 0x7ea, 0xa00, 0x0, 0x145aa9b37b9329ff)
        /root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/cache.go:337 +0x136
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*CacheLoader).Load.func1(0xc83bfc18b8, 0xc83bfc1a28, 0xc842343e80, 0x0, 0x0)
        /root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/cache.go:524 +0x7f3
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*CacheLoader).Load(0xc83bfc1a28, 0xc842343e80, 0x0, 0x0)
        /root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/cache.go:491 +0xcf
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).reloadCache(0xc843b378c0, 0x0, 0x0)
        /root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:715 +0x3b1
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).Open(0xc843b378c0, 0x0, 0x0)
        /root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:161 +0x1c3
github.com/influxdata/influxdb/tsdb.(*Shard).Open.func1(0xc832dab0e0, 0x0, 0x0)
        /root/go/src/github.com/influxdata/influxdb/tsdb/shard.go:165 +0x223
github.com/influxdata/influxdb/tsdb.(*Shard).Open(0xc832dab0e0, 0x0, 0x0)
        /root/go/src/github.com/influxdata/influxdb/tsdb/shard.go:145 +0x3d
github.com/influxdata/influxdb/tsdb.(*Store).loadShards.func1(0xc820144a40, 0xc8201c40f0, 0xc8201f43c0, 0xc8201df9a0, 0xc8201fa014, 0xe, 0xc820357fa3, 0x7, 0xc8206d223b, 0x3)
        /root/go/src/github.com/influxdata/influxdb/tsdb/store.go:177 +0x6ec
created by github.com/influxdata/influxdb/tsdb.(*Store).loadShards
        /root/go/src/github.com/influxdata/influxdb/tsdb/store.go:185 +0xa2e

[jackzampolin]

@josselin-c I would imagine that this is being caused by too many series in the database. As the instance tries to load all the tag values into memory, it runs out causing the OOM kill.

Do you have any estimate for how many series are in the dataset in question? I would also recomend looking at our technical paper on performance tuning

[daviesalex]

This may help: #6952

[josselin-c]

@jackzampolin From what I recall, the OOM only triggered once. After it triggered, influxdb was unable to restart and keept crashing during data reload with the stacktrace above. During restart, OOM was not triggered.

[daviesalex]

The PR I referenced "also fixes a panic that could occur to duplicate data stored within a block"

I have no idea if it is that panic or not, but that all I was thinking... perhaps worth a try?

[jwilder]

Fixed via bc76048