Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TSM1 Engine - panic: runtime error: index out of range - Values.MinTime #4444

Closed
syepes opened this issue Oct 14, 2015 · 5 comments
Closed

TSM1 Engine - panic: runtime error: index out of range - Values.MinTime #4444

syepes opened this issue Oct 14, 2015 · 5 comments
Assignees
@jwilder
Labels
area/tsm

Comments

@syepes
Copy link

syepes commented Oct 14, 2015

Hello,

After starting from scratch InfluxDB (87d99fb) after around 24h's I have got the following panic.

2015/10/13 23:13:51 InfluxDB starting, version 0.9, branch master, commit 87d99fb, built 1444754119
2015/10/13 23:13:51 Go version go1.5.1, GOMAXPROCS set to 8
2015/10/13 23:13:51 Using configuration at: /opt/influxdb/influxd.conf
[metastore] 2015/10/13 23:13:51 Using data dir: /opt/graphite/storage/influxdb/meta
...
[continuous_querier] 2015/10/14 09:22:16 wrote 56 point(s) to ESXi.raw
[tsm1wal] 2015/10/14 09:22:45 memory flush of /opt/graphite/storage/influxdb/data/ESXi/raw/4 with 8679 keys and 359602 total values of 5753632 bytes
[tsm1] 2015/10/14 09:22:46 rewriting index file /opt/graphite/storage/influxdb/data/ESXi/raw/4/0001371.tsm1 with /opt/graphite/storage/influxdb/data/ESXi/raw/4/0001379.tsm1
[tsm1] 2015/10/14 09:22:46 rewriting index file /opt/graphite/storage/influxdb/data/ESXi/raw/4/0001377.tsm1 with /opt/graphite/storage/influxdb/data/ESXi/raw/4/0001380.tsm1
panic: runtime error: index out of range

goroutine 62062 [running]:
github.com/influxdb/influxdb/tsdb/engine/tsm1.Values.MinTime(0xc863a91e80, 0x0, 0x0, 0x140cfb7e37d14800)
        /opt/gcollectors/src/go/src/github.com/influxdb/influxdb/tsdb/engine/tsm1/encoding.go:65 +0x75
github.com/influxdb/influxdb/tsdb/engine/tsm1.(*Engine).DecodeAndCombine(0xc846484a20, 0xc863a91e80, 0x0, 0x0, 0x9d5eec64e, 0x2d, 0x89f69bd, 0xc838f60000, 0x0, 0x4e20, ...)
        /opt/gcollectors/src/go/src/github.com/influxdb/influxdb/tsdb/engine/tsm1/tsm1.go:1668 +0x6bd
github.com/influxdb/influxdb/tsdb/engine/tsm1.(*Engine).rewriteFile(0xc846484a20, 0xc8621de9c0, 0xc8622c8060, 0x0, 0x0)
        /opt/gcollectors/src/go/src/github.com/influxdb/influxdb/tsdb/engine/tsm1/tsm1.go:1111 +0x208f
github.com/influxdb/influxdb/tsdb/engine/tsm1.(*Engine).Write.func3(0xc846484a20, 0xc863cadf40, 0xc850051480, 0xc863cadf50, 0xc8621de9c0, 0xc8622c8060)
        /opt/gcollectors/src/go/src/github.com/influxdb/influxdb/tsdb/engine/tsm1/tsm1.go:415 +0x3f
created by github.com/influxdb/influxdb/tsdb/engine/tsm1.(*Engine).Write
        /opt/gcollectors/src/go/src/github.com/influxdb/influxdb/tsdb/engine/tsm1/tsm1.go:421 +0x95c
goroutine 1 [chan receive, 607 minutes]:
main.(*Main).Run(0xc8201f7f10, 0xc82008a010, 0x1, 0x1, 0x0, 0x0)
        /opt/gcollectors/src/go/src/github.com/influxdb/influxdb/cmd/influxd/main.go:96 +0xa39
main.main()
        /opt/gcollectors/src/go/src/github.com/influxdb/influxdb/cmd/influxd/main.go:46 +0x29b

goroutine 17 [syscall, 607 minutes, locked to thread]:
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1

goroutine 20 [syscall, 607 minutes]:
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
        /usr/local/go/src/os/signal/signal_unix.go:28 +0x37
goroutine 37 [select]:
github.com/hashicorp/raft.(*Raft).leaderLoop(0xc8201ca000)
        /opt/gcollectors/src/go/src/github.com/hashicorp/raft/raft.go:849 +0xcce
github.com/hashicorp/raft.(*Raft).runLeader(0xc8201ca000)
        /opt/gcollectors/src/go/src/github.com/hashicorp/raft/raft.go:821 +0x936
github.com/hashicorp/raft.(*Raft).run(0xc8201ca000)
        /opt/gcollectors/src/go/src/github.com/hashicorp/raft/raft.go:602 +0xbb
github.com/hashicorp/raft.(*Raft).(github.com/hashicorp/raft.run)-fm()
        /opt/gcollectors/src/go/src/github.com/hashicorp/raft/raft.go:252 +0x20
github.com/hashicorp/raft.(*raftState).goFunc.func1(0xc8201ca000, 0xc820122be0)
        /opt/gcollectors/src/go/src/github.com/hashicorp/raft/state.go:152 +0x44
created by github.com/hashicorp/raft.(*raftState).goFunc

# ls - /opt/graphite/storage/influxdb/data/ESXi/raw/4/00013*
-rw-r--r--  1 root  influxd   3.5M Oct 14 08:55 /opt/graphite/storage/influxdb/data/ESXi/raw/4/0001300.tsm1
-rw-r--r--  1 root  influxd    11M Oct 14 08:55 /opt/graphite/storage/influxdb/data/ESXi/raw/4/0001301.tsm1
-rw-r--r--  1 root  influxd   3.9M Oct 14 08:55 /opt/graphite/storage/influxdb/data/ESXi/raw/4/0001302.tsm1
-rw-r--r--  1 root  influxd    11M Oct 14 08:55 /opt/graphite/storage/influxdb/data/ESXi/raw/4/0001303.tsm1
-rw-r--r--  1 root  influxd    11M Oct 14 08:58 /opt/graphite/storage/influxdb/data/ESXi/raw/4/0001304.tsm1
-rw-r--r--  1 root  influxd   120M Oct 14 08:59 /opt/graphite/storage/influxdb/data/ESXi/raw/4/0001305.tsm1
-rw-r--r--  1 root  influxd   9.0M Oct 14 09:01 /opt/graphite/storage/influxdb/data/ESXi/raw/4/0001321.tsm1
-rw-r--r--  1 root  influxd    11M Oct 14 09:01 /opt/graphite/storage/influxdb/data/ESXi/raw/4/0001322.tsm1
-rw-r--r--  1 root  influxd   688K Oct 14 09:01 /opt/graphite/storage/influxdb/data/ESXi/raw/4/0001323.tsm1
-rw-r--r--  1 root  influxd   1.1M Oct 14 09:02 /opt/graphite/storage/influxdb/data/ESXi/raw/4/0001329.tsm1
-rw-r--r--  1 root  influxd   9.8M Oct 14 09:02 /opt/graphite/storage/influxdb/data/ESXi/raw/4/0001330.tsm1
-rw-r--r--  1 root  influxd    11M Oct 14 09:02 /opt/graphite/storage/influxdb/data/ESXi/raw/4/0001331.tsm1
-rw-r--r--  1 root  influxd   120M Oct 14 09:05 /opt/graphite/storage/influxdb/data/ESXi/raw/4/0001332.tsm1
-rw-r--r--  1 root  influxd    11M Oct 14 09:08 /opt/graphite/storage/influxdb/data/ESXi/raw/4/0001337.tsm1
-rw-r--r--  1 root  influxd   2.8M Oct 14 09:08 /opt/graphite/storage/influxdb/data/ESXi/raw/4/0001340.tsm1
-rw-r--r--  1 root  influxd    10M Oct 14 09:08 /opt/graphite/storage/influxdb/data/ESXi/raw/4/0001341.tsm1
-rw-r--r--  1 root  influxd   3.1M Oct 14 09:10 /opt/graphite/storage/influxdb/data/ESXi/raw/4/0001342.tsm1
-rw-r--r--  1 root  influxd    10M Oct 14 09:10 /opt/graphite/storage/influxdb/data/ESXi/raw/4/0001343.tsm1
-rw-r--r--  1 root  influxd    10M Oct 14 09:12 /opt/graphite/storage/influxdb/data/ESXi/raw/4/0001344.tsm1
-rw-r--r--  1 root  influxd   921K Oct 14 09:15 /opt/graphite/storage/influxdb/data/ESXi/raw/4/0001368.tsm1
-rw-r--r--  1 root  influxd   9.8M Oct 14 09:15 /opt/graphite/storage/influxdb/data/ESXi/raw/4/0001369.tsm1
-rw-r--r--  1 root  influxd    10M Oct 14 09:15 /opt/graphite/storage/influxdb/data/ESXi/raw/4/0001370.tsm1
-rw-r--r--  1 root  influxd   1.1M Oct 14 09:15 /opt/graphite/storage/influxdb/data/ESXi/raw/4/0001371.tsm1
-rw-r--r--  1 root  influxd    10M Oct 14 09:16 /opt/graphite/storage/influxdb/data/ESXi/raw/4/0001374.tsm1
-rw-r--r--  1 root  influxd   9.9M Oct 14 09:16 /opt/graphite/storage/influxdb/data/ESXi/raw/4/0001375.tsm1
-rw-r--r--  1 root  influxd   9.9M Oct 14 09:18 /opt/graphite/storage/influxdb/data/ESXi/raw/4/0001376.tsm1
-rw-r--r--  1 root  influxd   138M Oct 14 09:21 /opt/graphite/storage/influxdb/data/ESXi/raw/4/0001377.tsm1
-rw-r--r--  1 root  influxd   9.9M Oct 14 09:20 /opt/graphite/storage/influxdb/data/ESXi/raw/4/0001378.tsm1
-rw-r--r--  1 root  influxd   698K Oct 14 09:22 /opt/graphite/storage/influxdb/data/ESXi/raw/4/0001379.tsm1
-rw-r--r--  1 root  influxd     0B Oct 14 09:22 /opt/graphite/storage/influxdb/data/ESXi/raw/4/0001379.tsm1.check
-rw-r--r--  1 root  influxd    50K Oct 14 09:22 /opt/graphite/storage/influxdb/data/ESXi/raw/4/0001380.tsm1
-rw-r--r--  1 root  influxd     0B Oct 14 09:22 /opt/graphite/storage/influxdb/data/ESXi/raw/4/0001380.tsm1.check

If you need more details don't hesitate in asking.

Regards,
Seabstian
@jwilder
Copy link
Contributor

jwilder commented Oct 16, 2015

This might be fixed by #4402

@beckettsean
Copy link
Contributor

closing as duplicate of #4365

@syepes syepes mentioned this issue Oct 17, 2015
Merged
3 tasks
@otoolep
Copy link
Contributor

otoolep commented Oct 17, 2015

Issue reported as still occurring.

@otoolep otoolep reopened this Oct 17, 2015
jwilder added a commit that referenced this issue Oct 19, 2015
@jwilder
Fix panic: runtime error: index out of range - Values.MinTime
ba73b1f 
When rewriting a tsm file, a panice on the Values slice could happen
if there were no values in the slice and the conditions of the rewrite
causes DecodeAndCombine to be called with the empty slice.  This could
happen is the sizes of the points new values was equal to
the MaxPointsInBlock config options and there were no future blocks after
the current one being written.

When this happens, DecodeAndCombine returns a zero length remaining values
slice which is passed back into DecodeAndCombine one last time.  In this case,
we now just return the original block since there is nothing new to combine.

Fixes #4444 #4365
@jwilder jwilder mentioned this issue Oct 19, 2015
Merged
@jwilder jwilder self-assigned this Oct 19, 2015
@syepes
Copy link
Author

syepes commented Oct 21, 2015

It's been now three days and InfluxDB is still running, I think the Values.MinTime issue has been finally resolved.

@otoolep
Copy link
Contributor

otoolep commented Oct 21, 2015

Thanks for letting us know @syepes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jwilder jwilder

Labels
area/tsm
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@jwilder @syepes @otoolep @beckettsean