After intense write load: "error compacting TSM files: ... exceeds max index entries"

[mark-rushakoff]

Bug report

System info:

OSX, InfluxDB from master @ 93745d9, default configuration, started with an empty ~/.influxdb folder.

Steps to reproduce:

Flood the system with writes in 20k batches, single series with single point, real-time nanosecond precision.

Expected behavior:

TSM compaction succeeds.

Actual behavior:

Queries seem to complete successfully, but seeing repeated messages about TSM compaction failing:

[tsm1] 2016/04/16 18:22:01 beginning full compaction of group 3, 4 TSM files
[tsm1] 2016/04/16 18:22:01 compacting full group (3) /Users/mr/.influxdb/data/b3k2m1mjvqqw_0/default/4/000000272-000000005.tsm (#0)
[tsm1] 2016/04/16 18:22:01 compacting full group (3) /Users/mr/.influxdb/data/b3k2m1mjvqqw_0/default/4/000000288-000000004.tsm (#1)
[tsm1] 2016/04/16 18:22:01 compacting full group (3) /Users/mr/.influxdb/data/b3k2m1mjvqqw_0/default/4/000000304-000000004.tsm (#2)
[tsm1] 2016/04/16 18:22:01 compacting full group (3) /Users/mr/.influxdb/data/b3k2m1mjvqqw_0/default/4/000000320-000000004.tsm (#3)
[tsm1] 2016/04/16 18:22:07 error compacting TSM files: key 'ctr,expid=b3k2m1mjvqqw#!#n' exceeds max index entries: 136940 > 65535
[tsm1] 2016/04/16 18:22:07 error compacting TSM files: key 'ctr,expid=b3k2m1mjvqqw#!#n' exceeds max index entries: 138240 > 65535
[tsm1] 2016/04/16 18:22:07 error compacting TSM files: key 'ctr,expid=b3k2m1mjvqqw#!#n' exceeds max index entries: 137240 > 65535
[tsm1] 2016/04/16 18:22:07 error compacting TSM files: key 'ctr,expid=b3k2m1mjvqqw#!#n' exceeds max index entries: 138620 > 65535
[tsm1] 2016/04/16 18:22:07 error compacting TSM files: key 'ctr,expid=b3k2m1mjvqqw#!#n' exceeds max index entries: 193000 > 65535
[tsm1] 2016/04/16 18:22:08 error compacting TSM files: key 'ctr,expid=b3k2m1mjvqqw#!#n' exceeds max index entries: 223200 > 65535

Additional info:

44MB TSM file sent out-of-band since it exceeds the 25MB upload limit.

/cc @jwilder