index file fd leak in tsi branch

[marchtea]

Bug report

I've using latest master branch 4957b3d
after merging changes in #8468

OS: centos 6
go version: 1.7.4

It appears that index file fd is leaking.

cat maps | grep delete
result:
7f83f0f87000-7f83f74e3000 r--s 00000000 08:04 24182797                   /data/influxdb/data/app_6/duration_7/2740/index/L4-00000155.tsi (deleted)
7f8406810000-7f8407974000 r--s 00000000 08:04 24182837                   /data/influxdb/data/app_6/duration_7/2740/index/L2-00000207.tsi (deleted)
7f8408714000-7f840b991000 r--s 00000000 08:04 24182821                   /data/influxdb/data/app_6/duration_7/2740/index/L3-00000196.tsi (deleted)
7f8419a04000-7f841a50d000 r--s 00000000 08:04 24182840                   /data/influxdb/data/app_6/duration_7/2740/index/L1-00000200.tsi (deleted)
7f8420be7000-7f8426bf3000 r--s 00000000 08:04 24182819                   /data/influxdb/data/app_6/duration_7/2740/index/L4-00000187.tsi (deleted)
7f842759a000-7f8428f52000 r--s 00000000 08:04 24182831                   /data/influxdb/data/app_6/duration_7/2740/index/L2-00000201.tsi (deleted)
7f8428f52000-7f842ab64000 r--s 00000000 08:04 24182825                   /data/influxdb/data/app_6/duration_7/2740/index/L2-00000195.tsi (deleted)
7f842ab64000-7f842dec4000 r--s 00000000 08:04 24182817                   /data/influxdb/data/app_6/duration_7/2740/index/L3-00000186.tsi (deleted)
7f842e9c3000-7f842fd98000 r--s 00000000 08:04 24182848                   /data/influxdb/data/app_6/duration_7/2740/index/L1-00000210.tsi (deleted)
7f842fd98000-7f8433807000 r--s 00000000 08:04 30294029                   /data/influxdb/data/app_6/duration_7/2740/index/L3-00000164.tsi (deleted)
7f843faec000-7f8441ab8000 r--s 00000000 08:04 30294021                   /data/influxdb/data/app_6/duration_7/2740/index/L2-00000143.tsi (deleted)
7f8441ffc000-7f8443b02000 r--s 00000000 08:04 24182829                   /data/influxdb/data/app_6/duration_7/2740/index/L2-00000197.tsi (deleted)
7f8443b02000-7f844517d000 r--s 00000000 08:04 24182828                   /data/influxdb/data/app_6/duration_7/2740/index/L2-00000191.tsi (deleted)
7f844517d000-7f8446668000 r--s 00000000 08:04 24182826                   /data/influxdb/data/app_6/duration_7/2740/index/L1-00000188.tsi (deleted)
7f8446668000-7f8449364000 r--s 00000000 08:04 24182809                   /data/influxdb/data/app_6/duration_7/2740/index/L3-00000179.tsi (deleted)
7f8458756000-7f845e52f000 r--s 00000000 08:04 30294019                   /data/influxdb/data/app_6/duration_7/2740/index/L4-00000140.tsi (deleted)
7f845ee91000-7f8461b9f000 r--s 00000000 08:04 24182796                   /data/influxdb/data/app_6/duration_7/2740/index/L3-00000154.tsi (deleted)

[marchtea]

@benbjohnson I've add some log and finally got the reason.
In log, it shows that index file is mmap twice and only munmap once.

map file: /data/influxdb/data/app_6/duration_7/2773/index/L1-00000308.tsi, 0x7f3b94575000
...
map file: /data/influxdb/data/app_6/duration_7/2773/index/L1-00000308.tsi, 0x7f39a82bc000
...
unmap: 0x7f39a82bc000

Then, I add stack info to show which cause the missing munmap.

map file: /data/influxdb/data/app_6/duration_7/2773/index/L1-00000308.tsi, 0x7f3b94575000
goroutine 62 [running]:
github.com/influxdata/influxdb/pkg/mmap.Map(0xc49c198040, 0x3f, 0x0, 0x0, 0x0, 0x0, 0x0)
        /Users/summer/code/go/workspace/src/github.com/influxdata/influxdb/pkg/mmap/mmap_unix.go:37 +0x2e8
github.com/influxdata/influxdb/tsdb/index/tsi1.mapIndexFileSeriesBlockFile(0xc49fe28000, 0xc44ee37c58, 0x200000003, 0xc4207ccf01, 0x902965, 0x0, 0x0)
        /Users/summer/code/go/workspace/src/github.com/influxdata/influxdb/tsdb/index/tsi1/series_block.go:975 +0x3f
github.com/influxdata/influxdb/tsdb/index/tsi1.mapIndexFileSeriesBlock(0xe270a0, 0xc49fe28000, 0xc44ee37ba0, 0x200000003, 0xc42001ea00, 0xc44ee37ba0, 0xc4207ccff0, 0x48bae4)
        /Users/summer/code/go/workspace/src/github.com/influxdata/influxdb/tsdb/index/tsi1/series_block.go:954 +0x1c8
github.com/influxdata/influxdb/tsdb/index/tsi1.(*LogFile).updateSeriesOffsets(0xc433916000, 0xe270a0, 0xc49fe28000, 0xc4aa9e8320, 0x2, 0x2, 0xc4207cd298, 0x0, 0x0)
        /Users/summer/code/go/workspace/src/github.com/influxdata/influxdb/tsdb/index/tsi1/log_file.go:864 +0x68
github.com/influxdata/influxdb/tsdb/index/tsi1.(*LogFile).CompactTo(0xc433916000, 0xe270a0, 0xc49fe28000, 0x2000000, 0x6, 0xdacc5e, 0x0, 0x0)
        /Users/summer/code/go/workspace/src/github.com/influxdata/influxdb/tsdb/index/tsi1/log_file.go:792 +0x3f4
github.com/influxdata/influxdb/tsdb/index/tsi1.(*Index).compactLogFile(0xc4203901c0, 0xc433916000)

And it shows the evil code : )

// mapIndexFileSeriesBlockFile memory-maps a file to a series block.
func mapIndexFileSeriesBlockFile(f *os.File) (*SeriesBlock, []byte, error) {
	// Open a read-only memory map of the existing data.
	data, err := mmap.Map(f.Name())
	if err != nil {
		return nil, nil, err
	}
	data = data[len(FileSignature):] // This changes data thus cause munmap a different address.