panic: runtime error: slice bounds out of range

[itdevon]

System Info:

Amazon Linux 2017.03
Influxdb 1.2.4

Steps to reproduce:
Start influxdb service

Actual behavior:
Service crashes and produces a "panic: runtime error: slice bounds out of range" error

Actions taken:
I tried removing the WAL files and started the service, still crashed with same error.

Additional info:

[I] 2017-06-28T15:53:22Z InfluxDB starting, version 1.2.4, branch master, commit 77909d7
[I] 2017-06-28T15:53:22Z Go version go1.7.4, GOMAXPROCS set to 8
[I] 2017-06-28T15:53:22Z Using configuration at: /etc/influxdb/influxdb.conf
[I] 2017-06-28T15:53:22Z Using data dir: /var/lib/influxdb/data service=store
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/_internal/monitor/206/000000006-000000003.tsm (#0) opened in 564.632µs engine=tsm1 service=filestore
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/_internal/monitor/200/000000008-000000004.tsm (#0) opened in 621.573µs engine=tsm1 service=filestore
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/_internal/monitor/202/000000008-000000004.tsm (#0) opened in 1.244024ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/_internal/monitor/203/000000009-000000002.tsm (#0) opened in 1.451694ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/database/autogen/92/000000465-000000007.tsm (#1) opened in 2.185365ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/_internal/monitor/201/000000008-000000004.tsm (#0) opened in 887.09µs engine=tsm1 service=filestore
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/_internal/monitor/205/000000009-000000002.tsm (#0) opened in 2.058882ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/database/autogen/172/000000448-000000005.tsm (#1) opened in 5.194196ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/database/autogen/92/000000897-000000007.tsm (#3) opened in 11.52912ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/database/autogen/92/000000897-000000006.tsm (#2) opened in 21.763287ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/database/autogen/92/000001121-000000005.tsm (#4) opened in 27.215368ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/_internal/monitor/206 database index loaded in 28.745968ms service=shard
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/_internal/monitor/206 opened in 30.148163ms service=store
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/database/autogen/52/000000027-000000002.tsm (#0) opened in 2.67826ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/_internal/monitor/200 database index loaded in 32.439155ms service=shard
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/_internal/monitor/200 opened in 36.203379ms service=store
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/_internal/monitor/205 database index loaded in 34.510112ms service=shard
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/_internal/monitor/205 opened in 39.480552ms service=store
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/_internal/monitor/202 database index loaded in 37.078952ms service=shard
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/_internal/monitor/203 database index loaded in 36.882654ms service=shard
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/database/autogen/188/000001276-000000004.tsm (#2) opened in 4.744614ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/_internal/monitor/203 opened in 40.560292ms service=store
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/_internal/monitor/202 opened in 44.12305ms service=store
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/database/autogen/172/000000448-000000004.tsm (#0) opened in 41.746899ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/_internal/monitor/201 database index loaded in 42.16561ms service=shard
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/_internal/monitor/201 opened in 45.688671ms service=store
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/_internal/monitor/198/000000008-000000004.tsm (#0) opened in 354.933µs engine=tsm1 service=filestore
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/database/autogen/31/000000011-000000002.tsm (#0) opened in 433.703µs engine=tsm1 service=filestore
[I] 2017-06-28T15:53:22Z reading file /var/lib/influxdb/wal/database/autogen/211/_00001.wal, size 584411 engine=tsm1 service=cacheloader
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/database/autogen/196/000001816-000000007.tsm (#3) opened in 10.157762ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/database/autogen/92/000000465-000000006.tsm (#0) opened in 61.517144ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/_internal/monitor/198 database index loaded in 18.241216ms service=shard
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/_internal/monitor/198 opened in 23.41327ms service=store
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/database/autogen/60/000000028-000000004.tsm (#0) opened in 754.612µs engine=tsm1 service=filestore
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/database/autogen/188/000001276-000000002.tsm (#0) opened in 36.24167ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/database/autogen/196/000001816-000000005.tsm (#1) opened in 34.076793ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/database/autogen/196/000001816-000000004.tsm (#0) opened in 35.802563ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/database/autogen/196/000001816-000000006.tsm (#2) opened in 47.669867ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/database/autogen/188/000001276-000000003.tsm (#1) opened in 59.404731ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:22Z reading file /var/lib/influxdb/wal/database/autogen/211/_00002.wal, size 18736 engine=tsm1 service=cacheloader
[I] 2017-06-28T15:53:22Z reading file /var/lib/influxdb/wal/database/autogen/211/_00007.wal, size 3783543 engine=tsm1 service=cacheloader
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/database/autogen/31 database index loaded in 152.439151ms service=shard
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/database/autogen/31 opened in 153.25196ms service=store
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/database/autogen/36/000000027-000000002.tsm (#0) opened in 669.697µs engine=tsm1 service=filestore
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/database/autogen/52 database index loaded in 236.183589ms service=shard
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/database/autogen/52 opened in 239.459493ms service=store
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/database/autogen/44/000000028-000000004.tsm (#0) opened in 788.399µs engine=tsm1 service=filestore
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/database/autogen/60 database index loaded in 260.265515ms service=shard
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/database/autogen/60 opened in 261.533689ms service=store
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/_internal/monitor/207 database index loaded in 47.091µs service=shard
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/_internal/monitor/207 opened in 299.61µs service=store
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/database/autogen/204/000000580-000000003.tsm (#2) opened in 10.529688ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/database/autogen/204/000000584-000000003.tsm (#3) opened in 11.498037ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/database/autogen/204/000000432-000000006.tsm (#0) opened in 27.088102ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/database/autogen/204/000000576-000000005.tsm (#1) opened in 40.362912ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/database/autogen/204/000000588-000000002.tsm (#4) opened in 57.630117ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/database/autogen/44 database index loaded in 252.028338ms service=shard
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/database/autogen/44 opened in 253.214559ms service=store
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/database/autogen/68/000000027-000000002.tsm (#0) opened in 601.701µs engine=tsm1 service=filestore
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/database/autogen/36 database index loaded in 339.235992ms service=shard
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/database/autogen/36 opened in 340.364187ms service=store
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/database/autogen/140/000000774-000000005.tsm (#0) opened in 25.9076ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:22Z /var/lib/influxdb/data/database/autogen/140/000000774-000000006.tsm (#1) opened in 29.14182ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:23Z reading file /var/lib/influxdb/wal/database/autogen/211/_00008.wal, size 781068 engine=tsm1 service=cacheloader
[I] 2017-06-28T15:53:23Z /var/lib/influxdb/data/database/autogen/68 database index loaded in 252.100205ms service=shard
[I] 2017-06-28T15:53:23Z /var/lib/influxdb/data/database/autogen/68 opened in 253.091006ms service=store
[I] 2017-06-28T15:53:23Z /var/lib/influxdb/data/_internal/monitor/199/000000008-000000004.tsm (#0) opened in 285.277µs engine=tsm1 service=filestore
[I] 2017-06-28T15:53:23Z /var/lib/influxdb/data/_internal/monitor/199 database index loaded in 5.50607ms service=shard
[I] 2017-06-28T15:53:23Z /var/lib/influxdb/data/_internal/monitor/199 opened in 6.243794ms service=store
[I] 2017-06-28T15:53:23Z reading file /var/lib/influxdb/wal/database/autogen/211/_00010.wal, size 3599084 engine=tsm1 service=cacheloader
[I] 2017-06-28T15:53:23Z /var/lib/influxdb/data/database/autogen/124/000001412-000000005.tsm (#0) opened in 9.836434ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:23Z /var/lib/influxdb/data/database/autogen/124/000001412-000000006.tsm (#1) opened in 9.989199ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:23Z /var/lib/influxdb/data/database/autogen/124/000001412-000000007.tsm (#2) opened in 15.224484ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:23Z /var/lib/influxdb/data/database/autogen/124/000001412-000000008.tsm (#3) opened in 16.501154ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:23Z reading file /var/lib/influxdb/wal/database/autogen/211/_00013.wal, size 1439719 engine=tsm1 service=cacheloader
[I] 2017-06-28T15:53:23Z reading file /var/lib/influxdb/wal/database/autogen/211/_00014.wal, size 447486 engine=tsm1 service=cacheloader
[I] 2017-06-28T15:53:23Z reading file /var/lib/influxdb/wal/database/autogen/211/_00019.wal, size 94922 engine=tsm1 service=cacheloader
[I] 2017-06-28T15:53:23Z reading file /var/lib/influxdb/wal/database/autogen/211/_00021.wal, size 73861 engine=tsm1 service=cacheloader
[I] 2017-06-28T15:53:23Z reading file /var/lib/influxdb/wal/database/autogen/211/_00022.wal, size 1993756 engine=tsm1 service=cacheloader
[I] 2017-06-28T15:53:24Z reading file /var/lib/influxdb/wal/database/autogen/211/_00023.wal, size 3283951 engine=tsm1 service=cacheloader
[I] 2017-06-28T15:53:24Z reading file /var/lib/influxdb/wal/database/autogen/211/_00025.wal, size 15689 engine=tsm1 service=cacheloader
[I] 2017-06-28T15:53:24Z reading file /var/lib/influxdb/wal/database/autogen/211/_00026.wal, size 1001248 engine=tsm1 service=cacheloader
[I] 2017-06-28T15:53:24Z reading file /var/lib/influxdb/wal/database/autogen/211/_00028.wal, size 94693 engine=tsm1 service=cacheloader
[I] 2017-06-28T15:53:24Z reading file /var/lib/influxdb/wal/database/autogen/211/_00031.wal, size 36799 engine=tsm1 service=cacheloader
[I] 2017-06-28T15:53:24Z reading file /var/lib/influxdb/wal/database/autogen/211/_00032.wal, size 1157082 engine=tsm1 service=cacheloader
[I] 2017-06-28T15:53:24Z reading file /var/lib/influxdb/wal/database/autogen/211/_00036.wal, size 3996073 engine=tsm1 service=cacheloader
[I] 2017-06-28T15:53:25Z reading file /var/lib/influxdb/wal/database/autogen/211/_00057.wal, size 4481 engine=tsm1 service=cacheloader
[I] 2017-06-28T15:53:25Z reading file /var/lib/influxdb/wal/database/autogen/211/_00060.wal, size 12870 engine=tsm1 service=cacheloader
[I] 2017-06-28T15:53:25Z reading file /var/lib/influxdb/wal/database/autogen/211/_00062.wal, size 3279 engine=tsm1 service=cacheloader
[I] 2017-06-28T15:53:25Z reading file /var/lib/influxdb/wal/database/autogen/211/_00063.wal, size 2262 engine=tsm1 service=cacheloader
[I] 2017-06-28T15:53:25Z reading file /var/lib/influxdb/wal/database/autogen/211/_00065.wal, size 6251 engine=tsm1 service=cacheloader
[I] 2017-06-28T15:53:25Z reading file /var/lib/influxdb/wal/database/autogen/211/_00066.wal, size 4551 engine=tsm1 service=cacheloader
[I] 2017-06-28T15:53:25Z reading file /var/lib/influxdb/wal/database/autogen/211/_00069.wal, size 5776 engine=tsm1 service=cacheloader
[I] 2017-06-28T15:53:25Z reading file /var/lib/influxdb/wal/database/autogen/211/_00084.wal, size 862 engine=tsm1 service=cacheloader
[I] 2017-06-28T15:53:25Z reading file /var/lib/influxdb/wal/database/autogen/211/_00085.wal, size 4986 engine=tsm1 service=cacheloader
[I] 2017-06-28T15:53:25Z reading file /var/lib/influxdb/wal/database/autogen/211/_00086.wal, size 0 engine=tsm1 service=cacheloader
[I] 2017-06-28T15:53:29Z /var/lib/influxdb/data/database/autogen/172 database index loaded in 7.281394467s service=shard
[I] 2017-06-28T15:53:29Z /var/lib/influxdb/data/database/autogen/172 opened in 7.326053771s service=store
[I] 2017-06-28T15:53:29Z /var/lib/influxdb/data/database/autogen/132/000001373-000000006.tsm (#0) opened in 13.851723ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:29Z /var/lib/influxdb/data/database/autogen/132/000001373-000000007.tsm (#1) opened in 14.560535ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:29Z /var/lib/influxdb/data/database/autogen/132/000001373-000000009.tsm (#3) opened in 18.539059ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:29Z /var/lib/influxdb/data/database/autogen/132/000001373-000000008.tsm (#2) opened in 19.483695ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:32Z /var/lib/influxdb/data/database/autogen/140 database index loaded in 9.328268639s service=shard
[I] 2017-06-28T15:53:32Z /var/lib/influxdb/data/database/autogen/140 opened in 9.357981355s service=store
[I] 2017-06-28T15:53:32Z /var/lib/influxdb/data/database/autogen/76/000000027-000000002.tsm (#0) opened in 668.361µs engine=tsm1 service=filestore
[I] 2017-06-28T15:53:32Z /var/lib/influxdb/data/database/autogen/76 database index loaded in 389.600593ms service=shard
[I] 2017-06-28T15:53:32Z /var/lib/influxdb/data/database/autogen/76 opened in 390.644542ms service=store
[I] 2017-06-28T15:53:32Z /var/lib/influxdb/data/database/autogen/84/000000030-000000002.tsm (#0) opened in 579.544µs engine=tsm1 service=filestore
[I] 2017-06-28T15:53:33Z /var/lib/influxdb/data/database/autogen/124 database index loaded in 9.844836841s service=shard
[I] 2017-06-28T15:53:33Z /var/lib/influxdb/data/database/autogen/124 opened in 9.861880893s service=store
[I] 2017-06-28T15:53:33Z reading file /var/lib/influxdb/wal/_internal/monitor/213/_00001.wal, size 46135 engine=tsm1 service=cacheloader
[I] 2017-06-28T15:53:33Z reading file /var/lib/influxdb/wal/_internal/monitor/213/_00018.wal, size 0 engine=tsm1 service=cacheloader
[I] 2017-06-28T15:53:33Z /var/lib/influxdb/data/_internal/monitor/213 database index loaded in 5.017503ms service=shard
[I] 2017-06-28T15:53:33Z /var/lib/influxdb/data/_internal/monitor/213 opened in 9.932438ms service=store
[I] 2017-06-28T15:53:33Z /var/lib/influxdb/data/_internal/monitor/212 database index loaded in 40.725µs service=shard
[I] 2017-06-28T15:53:33Z /var/lib/influxdb/data/_internal/monitor/212 opened in 266.25µs service=store
[I] 2017-06-28T15:53:33Z /var/lib/influxdb/data/database/autogen/116/000001382-000000004.tsm (#0) opened in 6.224081ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:33Z /var/lib/influxdb/data/database/autogen/116/000001382-000000005.tsm (#1) opened in 7.065551ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:33Z /var/lib/influxdb/data/database/autogen/116/000001382-000000007.tsm (#3) opened in 8.118748ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:33Z /var/lib/influxdb/data/database/autogen/116/000001382-000000006.tsm (#2) opened in 9.631865ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:33Z /var/lib/influxdb/data/database/autogen/84 database index loaded in 491.800285ms service=shard
[I] 2017-06-28T15:53:33Z /var/lib/influxdb/data/database/autogen/84 opened in 492.683741ms service=store
[I] 2017-06-28T15:53:33Z /var/lib/influxdb/data/database/autogen/164/000001059-000000006.tsm (#2) opened in 7.786722ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:33Z /var/lib/influxdb/data/database/autogen/164/000001059-000000005.tsm (#1) opened in 8.33082ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:33Z /var/lib/influxdb/data/database/autogen/164/000001059-000000004.tsm (#0) opened in 8.519901ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:36Z /var/lib/influxdb/data/database/autogen/188 database index loaded in 14.193233463s service=shard
[I] 2017-06-28T15:53:36Z /var/lib/influxdb/data/database/autogen/188 opened in 14.255247668s service=store
[I] 2017-06-28T15:53:36Z /var/lib/influxdb/data/database/autogen/100/000001439-000000004.tsm (#1) opened in 4.630863ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:36Z /var/lib/influxdb/data/database/autogen/100/000001439-000000003.tsm (#0) opened in 4.796583ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:36Z /var/lib/influxdb/data/database/autogen/100/000001439-000000005.tsm (#2) opened in 7.359205ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:36Z /var/lib/influxdb/data/database/autogen/100/000001439-000000006.tsm (#3) opened in 8.428967ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:37Z /var/lib/influxdb/data/database/autogen/211 database index loaded in 12.235583411s service=shard
[I] 2017-06-28T15:53:37Z /var/lib/influxdb/data/database/autogen/211 opened in 14.876569292s service=store
[I] 2017-06-28T15:53:37Z /var/lib/influxdb/data/database/autogen/108/000001455-000000003.tsm (#0) opened in 15.919805ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:37Z /var/lib/influxdb/data/database/autogen/108/000001455-000000004.tsm (#1) opened in 7.107152ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:37Z /var/lib/influxdb/data/database/autogen/108/000001455-000000005.tsm (#2) opened in 10.976926ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:37Z /var/lib/influxdb/data/database/autogen/108/000001455-000000006.tsm (#3) opened in 11.656831ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:39Z /var/lib/influxdb/data/database/autogen/164 database index loaded in 6.290317738s service=shard
[I] 2017-06-28T15:53:39Z /var/lib/influxdb/data/database/autogen/164 opened in 6.299343156s service=store
[I] 2017-06-28T15:53:39Z /var/lib/influxdb/data/_internal/monitor/208 database index loaded in 71.775µs service=shard
[I] 2017-06-28T15:53:39Z /var/lib/influxdb/data/_internal/monitor/208 opened in 325.203µs service=store
[I] 2017-06-28T15:53:39Z /var/lib/influxdb/data/_internal/monitor/209 database index loaded in 32.875µs service=shard
[I] 2017-06-28T15:53:39Z /var/lib/influxdb/data/_internal/monitor/209 opened in 284.268µs service=store
[I] 2017-06-28T15:53:39Z /var/lib/influxdb/data/_internal/monitor/210 database index loaded in 31.896µs service=shard
[I] 2017-06-28T15:53:39Z /var/lib/influxdb/data/_internal/monitor/210 opened in 222.719µs service=store
[I] 2017-06-28T15:53:39Z /var/lib/influxdb/data/database/autogen/156/000000909-000000004.tsm (#2) opened in 2.003473ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:39Z /var/lib/influxdb/data/database/autogen/156/000000909-000000002.tsm (#0) opened in 8.697202ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:39Z /var/lib/influxdb/data/database/autogen/156/000000909-000000003.tsm (#1) opened in 10.705343ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:40Z /var/lib/influxdb/data/database/autogen/196 database index loaded in 17.961583379s service=shard
[I] 2017-06-28T15:53:40Z /var/lib/influxdb/data/database/autogen/196 opened in 18.013315134s service=store
[I] 2017-06-28T15:53:40Z /var/lib/influxdb/data/database/autogen/148/000000841-000000004.tsm (#0) opened in 7.039096ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:40Z /var/lib/influxdb/data/database/autogen/148/000000841-000000005.tsm (#1) opened in 7.962432ms engine=tsm1 service=filestore
[I] 2017-06-28T15:53:40Z /var/lib/influxdb/data/database/autogen/116 database index loaded in 7.821612635s service=shard
[I] 2017-06-28T15:53:40Z /var/lib/influxdb/data/database/autogen/116 opened in 7.831635994s service=store
[I] 2017-06-28T15:53:41Z /var/lib/influxdb/data/database/autogen/132 database index loaded in 11.87554993s service=shard
[I] 2017-06-28T15:53:41Z /var/lib/influxdb/data/database/autogen/132 opened in 11.89552879s service=store
[I] 2017-06-28T15:53:42Z /var/lib/influxdb/data/database/autogen/100 database index loaded in 5.716022131s service=shard
[I] 2017-06-28T15:53:42Z /var/lib/influxdb/data/database/autogen/100 opened in 5.724894881s service=store
[I] 2017-06-28T15:53:43Z /var/lib/influxdb/data/database/autogen/156 database index loaded in 3.572796245s service=shard
[I] 2017-06-28T15:53:43Z /var/lib/influxdb/data/database/autogen/156 opened in 3.584259695s service=store
[I] 2017-06-28T15:53:43Z /var/lib/influxdb/data/database/autogen/148 database index loaded in 2.865746032s service=shard
[I] 2017-06-28T15:53:43Z /var/lib/influxdb/data/database/autogen/148 opened in 2.874177399s service=store
[I] 2017-06-28T15:53:44Z /var/lib/influxdb/data/database/autogen/92 database index loaded in 21.788612363s service=shard
[I] 2017-06-28T15:53:44Z /var/lib/influxdb/data/database/autogen/92 opened in 21.852784132s service=store
[I] 2017-06-28T15:53:44Z /var/lib/influxdb/data/database/autogen/108 database index loaded in 7.116319389s service=shard
[I] 2017-06-28T15:53:44Z /var/lib/influxdb/data/database/autogen/108 opened in 7.14463985s service=store
[I] 2017-06-28T15:53:44Z /var/lib/influxdb/data/database/autogen/204 database index loaded in 22.114671752s service=shard
[I] 2017-06-28T15:53:44Z /var/lib/influxdb/data/database/autogen/204 opened in 22.184015653s service=store
[I] 2017-06-28T15:53:44Z opened service service=subscriber
[I] 2017-06-28T15:53:44Z Starting monitor system service=monitor
[I] 2017-06-28T15:53:44Z 'build' registered for diagnostics monitoring service=monitor
[I] 2017-06-28T15:53:44Z 'runtime' registered for diagnostics monitoring service=monitor
[I] 2017-06-28T15:53:44Z 'network' registered for diagnostics monitoring service=monitor
[I] 2017-06-28T15:53:44Z 'system' registered for diagnostics monitoring service=monitor
[I] 2017-06-28T15:53:44Z Starting precreation service with check interval of 10m0s, advance period of 30m0s service=shard-precreation
[I] 2017-06-28T15:53:44Z Starting snapshot service service=snapshot
[I] 2017-06-28T15:53:44Z Starting continuous query service service=continuous_querier
[I] 2017-06-28T15:53:44Z Starting HTTP service service=httpd
[I] 2017-06-28T15:53:44Z Authentication enabled:false service=httpd
[I] 2017-06-28T15:53:44Z Storing statistics in database '_internal' retention policy 'monitor', at interval 10s service=monitor
[I] 2017-06-28T15:53:44Z Listening on HTTP:[::]:8086 service=httpd
[I] 2017-06-28T15:53:44Z Starting retention policy enforcement service with check interval of 30m0s service=retention
[I] 2017-06-28T15:53:44Z added new subscription for _internal monitor service=subscriber
[I] 2017-06-28T15:53:44Z added new subscription for database autogen service=subscriber
[I] 2017-06-28T15:53:44Z Started listening on UDP: 0.0.0.0:25826 service=udp
[I] 2017-06-28T15:53:44Z Sending usage statistics to usage.influxdata.com
[I] 2017-06-28T15:53:44Z Listening for signals
[I] 2017-06-28T15:53:44Z Failed to parse points: unable to parse 'hostname-98-224.hostname
T�3�A�g

                                                                                                    systemfile_handles': missing fields

unable to parse 'gaugefile_handles_max���WA
T�3�A��!file_handles_allocatedunused
T�B=_�Kaggregation
cpu-sucpu idlew/�cpu-average.��': missing fields
unable to parse 'stealc
cpu-sum3
softirqVminterruptcpu-average nice wait�
system/
cpu-sum usern�cpu-average��
cpu-sum
systemx� wait
�cpu-average
softirq': missing fields
unable to parse '�
cpu-sum nice
T�B=d� interfaceveth30bf253if_packetsQ��Q�l
T�B=d�Fif_octets�;�p6t�
T�B=d��if_errors
T�B=d��if_dropped
T�B=d��vethf305b97if_packetsy�,�
T�B=d�if_octetst?�': missing fields
unable to parse '�Zs
T�B=d�if_errors
T�B=d�if_dropped
/ T�B=e�loif_packets
T�B=e ^if_octetsb&b&
T�B=e
�if_errors': missing fields service=udp
[
panic: runtime error: slice bounds out of range

goroutine 468 [running]:
panic(0xa09d20, 0xc4200100d0)
/usr/local/go/src/runtime/panic.go:500 +0x1a1
github.com/influxdata/influxdb/models.scanLine(0xc478241180, 0x53a, 0x53a, 0x4f4, 0x3, 0xc4641d1700, 0x2, 0x4)
/root/go/src/github.com/influxdata/influxdb/models/points.go:1021 +0x8e
github.com/influxdata/influxdb/models.ParsePointsWithPrecision(0xc478241180, 0x53a, 0x53a, 0xed0e5c809, 0x1548d4d2, 0xe59940, 0xab962a, 0x1, 0xc46f2d5bf0, 0x0, ...)
/root/go/src/github.com/influxdata/influxdb/models/points.go:264 +0x13b
github.com/influxdata/influxdb/services/udp.(*Service).parser(0xc4201a0900)
/root/go/src/github.com/influxdata/influxdb/services/udp/service.go:214 +0x1f1
created by github.com/influxdata/influxdb/services/udp.(*Service).Open
/root/go/src/github.com/influxdata/influxdb/services/udp/service.go:119 +0x54110.

[e-dard]

I suspect that this will still be an issue in 1.3.0 based on the lack of activity in that part of points.go.

@devon21 does this panic occur as soon as you start up without any writes coming in, or are you writing to the DB as it's started up? I only ask as it looks like the server starts up and reads all the local files to build the indexes etc but then panics when it receives the write with the dodgy point.

Speaking of which—is it possible to sanitise the data you're sending to the db? Nicely formatted points would look like measurement,tagkey=tagvalue field=1.0,field2=33i for example.

[itdevon]

@e-dard Once influx is done loading, I see data coming in and that's when it goes into a panic mode. We use telegraf and I just updated to the latest, hoping it will help.. No luck.

[e-dard]

@devon21 what telegraf plugins are you using, i.e., what data is it that's being collected and written to Influx? The easiest way for us to fix this issue is to reproduce it, either with the point that panics the server, or by running an environment that will generate that point.

[itdevon]

Telegraf plugins:

[global_tags]
[agent]
[[outputs.influxdb]]
[[inputs.cpu]]
[[inputs.couchbase]]
[[inputs.net]]
[[inputs.cloudwatch]]
[[inputs.cloudwatch.metrics]]
[[inputs.cloudwatch.metrics.dimensions]]
[[inputs.redis]]

Let me know if you need anything else.

Last part of the influx log, right before if panics. Let me know if you want the full log..

[I] 2017-06-28T15:53:45Z Failed to parse points: unable to parse 'hostname-97-195.hostname
T�3WXo
ps_count@��@ processes java
T�3WX Qps_disk_octets�W��z|[
T�3WX
zps_disk_ops�P�L
T�3WX�ps_pagefaults�cZ
T�3WXps_stacksize;����B
T�3WX:ps_cputime{�.�B�<
ps_state T�3WW�A
stopped
T�3WZX�fork_ratet��
T�7?�@�
systemfile_handles': missing fields
unable to parse 'gaugefile_handles_used��@
T�7?���file_handles_max8IA
T�7?��!file_handles_allocatedunused
T�BV�aggregation
� cpu-sucpu idleHl�cpu-average�
cpu-sum
softirq��cpu-averageЛ
cpu-suminterrupt nicecpu-average
cpu-sum waite�cpu-average
�': missing fields
unable to parse 'steal-�interrupt userd��
system#
cpu-sum user(Sy
system': missing fields
unable to parse 'stealn�
T�BV�d load load!������@h�z�G @��Q��
@
T�BV�|cpucpu userha': missing fields service=udp
[I] 2017-06-28T15:53:45Z Failed to parse points: unable to parse 'hostname-97-195.hostname
T�BVҊ
cpucpu user|��
T�BVҒ�3f�
T�BVҚinterfaceveth66d97c1if_packets����k
T�BVҜcpucpu user`�
T�BVҡinterfaceveth66d97c1if_octetsL��k���b
T�BVҡcpucpu userd��
T�BVҩ�6J��
T�BVҫinterfaceveth66d97c1if_errors
T�BVҸcpucpu
system�
T�BVҲ�0�U
T�BV�nG userk�W
T�BVҿ�interfaceveth225d2f3if_octetsnZ��Au�|
T�BVҰ�veth66d97c1if_dropped
T�BVҼcpucpu
system՞
T�BV���interfaceveth225d2f3if_errors
T�BV��cpucpu
systemD
T�BV��Einterfaceveth225d2f3if_dropped
T�BVҺEif_packets�����
T�BV��cpucpu
systemm�
T�BV��o6�r
T�BV��@7*�': missing fields service=udp
panic: runtime error: slice bounds out of range

goroutine 468 [running]:
panic(0xa09d20, 0xc4200100d0)

[danielnelson]

@devon21 Could you add a file output along with your influxdb output and run telegraf until you receive this error? We can then check the file to see if it contains corrupted text.

[itdevon]

The influx service crashes before telegraf has a chance to write the file.

[danielnelson]

Nothing is written to the file at all? This is somewhat surprising to me but I guess you could just remove the influxdb output temporarily.

[itdevon]

Still having the same issue... Influx service crashes to quickly..

[itdevon]

Is there a workaround until this get fixed?

[danielnelson]

Can you try disabling the influxdb output in Telegraf and write only to a file output, and then check the output file for any corruption?

[danielnelson]

@devon21 Can you also try switching to use HTTP instead of UDP?

[jwilder]

@devon21 Can you enable?

[http]
  write-tracing = true

It will dump out the raw write data to the logs.

[itdevon]

I did this and the logs have this now..

(end of log)
X^@A^@ ^@^L^@^@^@^O^@^@^@^@^@^B^@^Hcpu^@^@^C^@^F4^@^@^D^@^Hcpu^@^@^E^@^Ksystem^@^@^F^@^O
^@^A^B^@^@^@^@^@^K8^@^H^@^L^VX^@A<94>^@^C^@^F2^@^@^F^@^O^@^A^B^@^@^@^@^@^Kp/^@^H^@^L^VX^@At^@^C^@^F1^@^@^E^@ wait^@^@^F^@^O^@^A^B^@^@^@^@^@^@^<96>^@^H^@^L^VX^@Al^@^C^@^F7^@^@^E^@^Ksystem^@^@^F^@^O^@^A^B^@^@^@^@^@^K^@^H^@^L^VX^@A^@^C^@^F2^@^@^E
^@ wait^@^@^F^@^O^@^A^B^@^@^@^@^@^@^Vn^@^H^@^L^VX^@AC^@^C^@^F4^@^@^F^@^O^@^A^B^@^@^@^@^@^@b(^@^H^@^L^VX^@AK^@^C^@^F3^@^@^F^@^O^@^A^B^@^@^@^@^@^@^V^W^@^H^@^L^VX^@A<^@^C^@^F7^@^@^F^@^O^@^A^B^@^@^@^@^@^@^<E7>^@^H^@^L^VX^@Am^@^C^@^F6^@^@^E^@^Ksystem^@^@^F^@^O^@^A^B^@
^@^@^@^@^M^@^H^@^L^VX^@A^Z^@^C^@^F5^@^@^E^@ wait^@^@^F^@^O^@^A^B^@^@^@^@^@^@*<8E>
^@^H^@^L^VX^@A<90>^@^C^@^F0^@^@^F^@^O^@^A^B^@^@^@^@^@^@^^5^@^H^@^L^VX^@A^H^@^C^@
^F6^@^@^F^@^O^@^A^B^@^@^@^@^@^@esc^@^H^@^L^VX^@A^@^C^@^F3^@^@^E^@ nice^@^@^F^@^O^@^A^B^@^@^@^@^@^@^@^W^@^H^@^L^VX^@A<80>^@^C^@^F1^@^@^F^@^O^@^A^B^@^@^@^@^@^@^@^^^@^H^@^L^VX
^@A^B<95>^@^C^@^F5^@^@^F^@^O^@^A^B^@^@^@^@^@^@^@^S^@^H^@^L^VX^@A^@^C^@^F0^@^@^F^@^O^@^A^B^@^@^@^@^@^@^@^H^@^H^@^L^VX^@A^Gg^@^C^@^F6^@^@^F^@^O^@^A^B^@^@^@^@^@^@^@': missing fields
unable to parse ^@^L^VX^@A&^@^C^@^F2^@^@^F^@^O^@^A^B^@^@^@^@^@^@^@^G^@^H^@^L^VX^@A&^Z^@^C^@^F0^@^@^E^@^Ninterrupt^@^@^F^@^O^@^A^B^@^@^@^@^@^@^@^@^@^H^@^L^VX^@A,^@^C^@^F1^@^@^F^@
^O^@^A^B^@^@^@^@^@^@^@^@^@^H^@^L^VX^@A^@^C^@^F2^@^@^F^@^O^@^A^B^@^@^@^@^@^@^@^@^@^H^@^L^VX^@AZ^@^C^@^F4^@^@^E^@ nice^@^@^F^@^O^@^A^B^@^@^@^@^@^@^@^Q^@^H^@^L^VX^@A8l^@^C^@^F3^@^@^E^@^Ninterrupt^@^@^F^@^O^@^A^B^@^@^@^@^@^@^@^@^@^H^@^L^VX^@AF^@^C^@^F6^@^@^F^@^O^@^A^B^@^@^@^@^@^@^@^@^@^H^@^L^VX^@A^K<90>^@^C^@^F7^@^@^E^@ nice^@^@^F^@^O^@^A^B^@^@^@^@^@^@^@^M^@^H^@^L
^VX^@APu^@^C^@^F0^@^@^E^@^Lsoftirq^@^@^F^@^O^@^A^B^@^@^@^@^@^A^@^H^@^L^VX^@A=v^@
^C^@^F4^@^@^E^@^Ninterrupt^@^@^F^@^O^@^A^B^@^@^@^@^@^@^@^@^@^H^@^L^VX^@AK<81>^@^C^@^F7^@^@^F^@^O
^@^A^B^@^@^@^@^@^@^@^@^@^H^@^L^VX^@Ac^M^@^C^@^F4^@^@^E^@^Lsoftirq^@^@^F^@^O^@^A^B^@^@^@^@^@^BL^@^H^@^L^VX^@A^^@^C^@^F3^@^@^F^@^O^@^A^B^@^@^@^@^@^Q^W ^@^H^@^L^VX^@Al^D^@^C^@^F6^@
^@^F^@^O^@^A^B^@^@^@^@^@^T<85>^B^@^H^@^L^VX^@AB^@^C^@^F5^@^@^E^@^Ninterrupt^@^@^F^@^O^@^A^B^@^@
^@^@^@^@^@^@^@^H^@^L^VX^@AU<87>^@^C^@^F1^@^@^E^@^Lsoftirq^@^@^F^@^O^@^A^B^@^@^@^@^@^D[': invalid field format service=udp
panic: runtime error: slice bounds out of range

goroutine 250 [running]:

[jwilder]

Ah. I missed that you are using the udp input. The setting I mentioned is for the HTTP plugin.

It looks like something is writing binary data to the UDP endpoint. That plugin expects line protocol.

[itdevon]

Are you talking about the plugin within Influxdb or Telegraf conf? The node that is sending traffic is using Telegraf. How would I stop Influxdb service from crashing?

[jwilder]

InfluxDB is is crashing because of bad input to the UDP plugin. It expects text line protocol, but something binary is getting sent triggering a panic. That is a bug as it should not be panicing and crashing. We need to figure out what it is sending so we can reproduce it and fix it.

The config setting I mentioned was for influxdb.

As a workaround, if you can switch telegraf to write to influxdb via HTTP instead of UDP, then the setting may help use identify what is causing the parser to panic.

[itdevon]

OK, I switched.. now that the service is not crashing, I see about 15 measurements corrupted. What do you need from me?

Ex.
WY��f�if_errors
WY��P�loif_dropped
WY��scpucpu user�B
WY��(�0�7
WY��^)interface
docker0if_packets<�@��>
WY��K�loif_errors

[danielnelson]

If you remove the corrupted measurements do they return under the http client?

[danielnelson]

This could be caused by influxdata/telegraf#3004.

[itdevon]

Once I switched to HTTP and removed the corrupt measurements, Influxdb is running smooth.

[e-dard]

@devon21 glad to hear it. This issue will stay open because the parser should never panic. We still need to find a reproducer for this issue so we can fix the parser.