Conversion to tsm fails with exceeds max index value

[pbooth]

Hello, I have a 0.9.6 influxdb that I have upgraded to 0.10.1-1
When I try to upgrade the shard in a 30GB DB I get the following:

2016/02/29 11:50:16.988594 Still Working: Completed Shards: 8/29 Points read/written: 306704235/306704235
2016/02/29 11:50:17.510485 Failed to convert    /data/fusionio/fio01/apps/tpeng/influxdb_lab15/data/prism/default/2210: Conversion of /data/fusionio/fio01/apps/tpeng/influxdb_lab15/data/prism/default/2210 failed: key 'latency_micros,cgroups=both,env=perf,instance=ampstestlab8,stor=nonpersist#!~#value' exceeds max index entries: 218400 > 65535

Is there something that I can with the bz shard to shrink it?

[jonseymour]

@pbooth it sounds like the TSM writer needs to deal with this condition as part of the conversion. @joelegasse will likely need to take a look, I think.

[jonseymour]

@pbooth one thing you might try is to reduce the -sz parameter to the influx_tsm command to some smaller value. provided you hit the file size limit before you hit the index entries limit, you should avoid the condition that causes the conversion to barf.

[joelegasse]

@pbooth Thank you for finding this. @jonseymour is right, we need to handle this in the conversion tool, and there's nothing you should need to do with your files.

This is the side effect of a design decision in TSM, specifically the tsm1 format's block index for a given key. Using the default values, it's limited to 65535 (a 16-bit index key) blocks of up to 1000 entries (values) per block, per TSM file. This means that over 65 million points would need to be stored for a single field in a single series in a single shard without going over the file limit for this to be an issue. As @jonseymour mentioned, usually the file-size limit is hit first, and even at the highest compaction, that index limit isn't hit.

I should have a fix merged shortly, and we're planning a 0.10.2 release in the near future that will also include some other fixes/optimizations for the conversion tool as well.