Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

influxdb 0.8.7: regular daily crashes due to rocksdb 3.5.1: upgrade rocksdb? #1251

Closed
apestov opened this issue Dec 19, 2014 · 1 comment
Closed

influxdb 0.8.7: regular daily crashes due to rocksdb 3.5.1: upgrade rocksdb? #1251

apestov opened this issue Dec 19, 2014 · 1 comment

Comments

@apestov
Copy link

apestov commented Dec 19, 2014

Hi,

I observe regular influxdb crashes -- it happens one or two times per day with the assert message below. Going down the corresponding code was fixed in rocksdb 3.6.1 --facebook/rocksdb#295 issue closed with facebook/rocksdb@2fb1fea commit.

Do you have any plans for upgrading rocksdb-3.5.1 to recent versions?

Info

OS is CentOS 6.6, Influxdb 0.8.7 has default settings, used as single instance (not a cluster)

influxdb: db/db_impl.cc:1881: rocksdb::Status rocksdb::DBImpl::RunManualCompaction(rocksdb::ColumnFamilyData*, int, int, uint32_t, const rocksdb::Slice*, const rocksdb::Slice*): Assertion `bg_compaction_scheduled_ == 0' failed.
SIGABRT: abort
PC=0x333d432625
signal arrived during cgo execution

goroutine 1631 [syscall]:
runtime.cgocall(0x4e5fba, 0x7fe537fa1998)
        /root/.gvm/gos/go1.3.1/src/pkg/runtime/cgocall.c:143 +0xe5 fp=0x7fe537fa1980 sp=0x7fe537fa1938
github.com/influxdb/rocksdb._Cfunc_rocksdb_compact_range(0x7fe5703dd170, 0x0, 0x0, 0x0, 0x0)
        github.com/influxdb/rocksdb/_obj/_cgo_defun.c:161 +0x31 fp=0x7fe537fa1998 sp=0x7fe537fa1980
github.com/influxdb/rocksdb.(*DB).CompactRange(0xc20803aab0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /root/gocodez/src/github.com/influxdb/rocksdb/db.go:311 +0xa9 fp=0x7fe537fa19f0 sp=0x7fe537fa1998
github.com/influxdb/influxdb/datastore/storage.RocksDB.Compact(0xc20803aab0, 0xc20803aa90, 0xc20803aaa0, 0xc20803aab8, 0xc20803aac0, 0xc20819b470, 0x2e)
        /root/gocodez/src/github.com/influxdb/influxdb/datastore/storage/rocksdb.go:92 +0x56 fp=0x7fe537fa1a60 sp=0x7fe537fa19f0
github.com/influxdb/influxdb/datastore/storage.(*RocksDB).Compact(0xc20819dbc0)
        <autogenerated>:46 +0x91 fp=0x7fe537fa1aa0 sp=0x7fe537fa1a60
github.com/influxdb/influxdb/datastore.(*Shard).executeDeleteQuery(0xc2081b6410, 0xc20af325a0, 0x7fe57b591250, 0x0, 0x0, 0x0)
        /root/gocodez/src/github.com/influxdb/influxdb/datastore/shard.go:330 +0x312 fp=0x7fe537fa1c58 sp=0x7fe537fa1aa0
github.com/influxdb/influxdb/datastore.(*Shard).Query(0xc2081b6410, 0xc20af325a0, 0x7fe57b591250, 0x0, 0x0, 0x0)
        /root/gocodez/src/github.com/influxdb/influxdb/datastore/shard.go:102 +0x1cf fp=0x7fe537fa1d50 sp=0x7fe537fa1c58
github.com/influxdb/influxdb/cluster.(*ShardData).Query(0xc208199860, 0xc20af325a0, 0xc209116a80)
        /root/gocodez/src/github.com/influxdb/influxdb/cluster/shard.go:327 +0xb0f fp=0x7fe537fa1f90 sp=0x7fe537fa1d50
runtime.goexit()
        /root/.gvm/gos/go1.3.1/src/pkg/runtime/proc.c:1445 fp=0x7fe537fa1f98 sp=0x7fe537fa1f90
created by github.com/influxdb/influxdb/coordinator.(*Coordinator).queryShards
        /root/gocodez/src/github.com/influxdb/influxdb/coordinator/coordinator.go:273 +0x340

goroutine 16 [IO wait]:
net.runtime_pollWait(0x7fe57b580608, 0x72, 0x0)
        /root/.gvm/gos/go1.3.1/src/pkg/runtime/netpoll.goc:146 +0x66
net.(*pollDesc).Wait(0xc2080268b0, 0x72, 0x0, 0x0)
        /root/.gvm/gos/go1.3.1/src/pkg/net/fd_poll_runtime.go:84 +0x46
net.(*pollDesc).WaitRead(0xc2080268b0, 0x0, 0x0)
        /root/.gvm/gos/go1.3.1/src/pkg/net/fd_poll_runtime.go:89 +0x42
net.(*netFD).accept(0xc208026850, 0xdbd518, 0x0, 0x7fe57b57d470, 0xb)
        /root/.gvm/gos/go1.3.1/src/pkg/net/fd_unix.go:419 +0x343
net.(*TCPListener).AcceptTCP(0xc20803a060, 0x8, 0x0, 0x0)
        /root/.gvm/gos/go1.3.1/src/pkg/net/tcpsock_posix.go:234 +0x5d
net.(*TCPListener).Accept(0xc20803a060, 0x0, 0x0, 0x0, 0x0)
        /root/.gvm/gos/go1.3.1/src/pkg/net/tcpsock_posix.go:244 +0x4b
net/http.(*Server).Serve(0xc208202120, 0x7fe57b57f7c0, 0xc20803a060, 0x0, 0x0)
        /root/.gvm/gos/go1.3.1/src/pkg/net/http/server.go:1698 +0x91
github.com/influxdb/influxdb/api/http.(*HttpServer).serveListener(0xc2080d34a0, 0x7fe57b57f7c0, 0xc20803a060, 0xc20803a3c0)
        /root/gocodez/src/github.com/influxdb/influxdb/api/http/api.go:202 +0xb9
github.com/influxdb/influxdb/api/http.(*HttpServer).Serve(0xc2080d34a0, 0x7fe57b57f7c0, 0xc20803a060)
        /root/gocodez/src/github.com/influxdb/influxdb/api/http/api.go:172 +0xcbd
github.com/influxdb/influxdb/api/http.(*HttpServer).ListenAndServe(0xc2080d34a0)
        /root/gocodez/src/github.com/influxdb/influxdb/api/http/api.go:91 +0x1a4
github.com/influxdb/influxdb/server.(*Server).ListenAndServe(0xc208044090, 0x0, 0x0)
        /root/gocodez/src/github.com/influxdb/influxdb/server/server.go:217 +0xa7b
main.start(0x0, 0x0)
        /root/gocodez/src/github.com/influxdb/influxdb/daemon/influxd.go:202 +0x101f
main.main()
        /root/gocodez/src/github.com/influxdb/influxdb/daemon/influxd.go:73 +0x1e

goroutine 19 [finalizer wait, 64 minutes]:
runtime.park(0x4fcc40, 0x14c8890, 0x14b2e89)
        /root/.gvm/gos/go1.3.1/src/pkg/runtime/proc.c:1369 +0x89
runtime.parkunlock(0x14c8890, 0x14b2e89)
        /root/.gvm/gos/go1.3.1/src/pkg/runtime/proc.c:1385 +0x3b
runfinq()
        /root/.gvm/gos/go1.3.1/src/pkg/runtime/mgc0.c:2644 +0xcf
runtime.goexit()
        /root/.gvm/gos/go1.3.1/src/pkg/runtime/proc.c:1445

goroutine 20 [syscall, 64 minutes]:
os/signal.loop()
        /root/.gvm/gos/go1.3.1/src/pkg/os/signal/signal_unix.go:21 +0x1e
created by os/signal.init·1
        /root/.gvm/gos/go1.3.1/src/pkg/os/signal/signal_unix.go:27 +0x32

goroutine 21 [chan receive, 64 minutes]:
code.google.com/p/log4go.ConsoleLogWriter.run(0xc208072000, 0x7fe57b57d5d0, 0xc20803a008)
        /root/gocodez/src/code.google.com/p/log4go/termlog.go:27 +0x79
created by code.google.com/p/log4go.NewConsoleLogWriter
        /root/gocodez/src/code.google.com/p/log4go/termlog.go:19 +0x68

goroutine 17 [syscall, 64 minutes]:
runtime.goexit()
        /root/.gvm/gos/go1.3.1/src/pkg/runtime/proc.c:1445

goroutine 22 [select]:
code.google.com/p/log4go.func·002()
        /root/gocodez/src/code.google.com/p/log4go/filelog.go:84 +0x8ac
created by code.google.com/p/log4go.NewFileLogWriter
        /root/gocodez/src/code.google.com/p/log4go/filelog.go:116 +0x2c4

goroutine 23 [chan receive]:
github.com/influxdb/influxdb/wal.(*WAL).processEntries(0xc2080267e0)
        /root/gocodez/src/github.com/influxdb/influxdb/wal/wal.go:252 +0x64
created by github.com/influxdb/influxdb/wal.NewWAL
        /root/gocodez/src/github.com/influxdb/influxdb/wal/wal.go:103 +0xa53

goroutine 33 [chan receive, 64 minutes]:
main.waitForSignals(0x7fe57b57f740, 0xc208044090)
        /root/gocodez/src/github.com/influxdb/influxdb/daemon/null_profiler.go:23 +0x14c
created by main.startProfiler
        /root/gocodez/src/github.com/influxdb/influxdb/daemon/null_profiler.go:15 +0x4b

goroutine 34 [IO wait, 64 minutes]:
net.runtime_pollWait(0x7fe57b580768, 0x72, 0x0)
        /root/.gvm/gos/go1.3.1/src/pkg/runtime/netpoll.goc:146 +0x66
net.(*pollDesc).Wait(0xc208026220, 0x72, 0x0, 0x0)
        /root/.gvm/gos/go1.3.1/src/pkg/net/fd_poll_runtime.go:84 +0x46
net.(*pollDesc).WaitRead(0xc208026220, 0x0, 0x0)
        /root/.gvm/gos/go1.3.1/src/pkg/net/fd_poll_runtime.go:89 +0x42
net.(*netFD).accept(0xc2080261c0, 0xdbd518, 0x0, 0x7fe57b57d470, 0xb)
        /root/.gvm/gos/go1.3.1/src/pkg/net/fd_unix.go:419 +0x343
net.(*TCPListener).AcceptTCP(0xc20803a030, 0xc2080ce068, 0x0, 0x0)
        /root/.gvm/gos/go1.3.1/src/pkg/net/tcpsock_posix.go:234 +0x5d
net.(*TCPListener).Accept(0xc20803a030, 0x0, 0x0, 0x0, 0x0)
        /root/.gvm/gos/go1.3.1/src/pkg/net/tcpsock_posix.go:244 +0x4b
net/http.(*Server).Serve(0xc2080aa240, 0x7fe57b57f7c0, 0xc20803a030, 0x0, 0x0)
        /root/.gvm/gos/go1.3.1/src/pkg/net/http/server.go:1698 +0x91
github.com/influxdb/influxdb/coordinator.func·007()
        /root/gocodez/src/github.com/influxdb/influxdb/coordinator/raft_server.go:579 +0x3a
created by github.com/influxdb/influxdb/coordinator.(*RaftServer).Serve
        /root/gocodez/src/github.com/influxdb/influxdb/coordinator/raft_server.go:583 +0x44c

goroutine 36 [select, 7 minutes]:
github.com/influxdb/influxdb/_vendor/raft.(*server).leaderLoop(0xc2080030e0)
        /root/gocodez/src/github.com/influxdb/influxdb/_vendor/raft/server.go:834 +0x6c5
github.com/influxdb/influxdb/_vendor/raft.(*server).loop(0xc2080030e0)
        /root/gocodez/src/github.com/influxdb/influxdb/_vendor/raft/server.go:609 +0x3d1
github.com/influxdb/influxdb/_vendor/raft.func·007()
        /root/gocodez/src/github.com/influxdb/influxdb/_vendor/raft/server.go:470 +0x5d
created by github.com/influxdb/influxdb/_vendor/raft.(*server).Start
        /root/gocodez/src/github.com/influxdb/influxdb/_vendor/raft/server.go:471 +0x3b8

goroutine 37 [select]:
github.com/influxdb/influxdb/coordinator.(*RaftServer).CompactLog(0xc2080d33f0)
        /root/gocodez/src/github.com/influxdb/influxdb/coordinator/raft_server.go:348 +0x332
created by github.com/influxdb/influxdb/coordinator.(*RaftServer).startRaft
        /root/gocodez/src/github.com/influxdb/influxdb/coordinator/raft_server.go:406 +0x512

goroutine 25 [select]:
github.com/influxdb/influxdb/coordinator.(*RaftServer).raftLeaderLoop(0xc2080d33f0, 0xc20818f340, 0xc20818f300)
        /root/gocodez/src/github.com/influxdb/influxdb/coordinator/raft_server.go:465 +0x34c
created by github.com/influxdb/influxdb/coordinator.(*RaftServer).raftEventHandler
        /root/gocodez/src/github.com/influxdb/influxdb/coordinator/raft_server.go:453 +0x268

goroutine 27 [IO wait, 63 minutes]:
net.runtime_pollWait(0x7fe57b5806b8, 0x72, 0x0)
        /root/.gvm/gos/go1.3.1/src/pkg/runtime/netpoll.goc:146 +0x66
net.(*pollDesc).Wait(0xc2082140d0, 0x72, 0x0, 0x0)
        /root/.gvm/gos/go1.3.1/src/pkg/net/fd_poll_runtime.go:84 +0x46
net.(*pollDesc).WaitRead(0xc2082140d0, 0x0, 0x0)
        /root/.gvm/gos/go1.3.1/src/pkg/net/fd_poll_runtime.go:89 +0x42
net.(*netFD).accept(0xc208214070, 0xdbd518, 0x0, 0x7fe57b57d470, 0xb)
        /root/.gvm/gos/go1.3.1/src/pkg/net/fd_unix.go:419 +0x343
net.(*TCPListener).AcceptTCP(0xc20808a000, 0xc2081fe020, 0x0, 0x0)
        /root/.gvm/gos/go1.3.1/src/pkg/net/tcpsock_posix.go:234 +0x5d
net.(*TCPListener).Accept(0xc20808a000, 0x0, 0x0, 0x0, 0x0)
        /root/.gvm/gos/go1.3.1/src/pkg/net/tcpsock_posix.go:244 +0x4b
github.com/influxdb/influxdb/coordinator.(*ProtobufServer).ListenAndServe(0xc2080ce4c0)
        /root/gocodez/src/github.com/influxdb/influxdb/coordinator/protobuf_server.go:68 +0x1c7
created by github.com/influxdb/influxdb/server.(*Server).ListenAndServe
        /root/gocodez/src/github.com/influxdb/influxdb/server/server.go:126 +0x2a2

goroutine 28 [select, 63 minutes]:
github.com/influxdb/influxdb/cluster.(*WriteBuffer).handleWrites(0xc208214000)
        /root/gocodez/src/github.com/influxdb/influxdb/cluster/write_buffer.go:75 +0xd3
created by github.com/influxdb/influxdb/cluster.NewWriteBuffer
        /root/gocodez/src/github.com/influxdb/influxdb/cluster/write_buffer.go:43 +0x286

goroutine 30 [IO wait, 63 minutes]:
net.runtime_pollWait(0x7fe57b580558, 0x72, 0x0)
        /root/.gvm/gos/go1.3.1/src/pkg/runtime/netpoll.goc:146 +0x66
net.(*pollDesc).Wait(0xc208214530, 0x72, 0x0, 0x0)
        /root/.gvm/gos/go1.3.1/src/pkg/net/fd_poll_runtime.go:84 +0x46
net.(*pollDesc).WaitRead(0xc208214530, 0x0, 0x0)
        /root/.gvm/gos/go1.3.1/src/pkg/net/fd_poll_runtime.go:89 +0x42
net.(*netFD).accept(0xc2082144d0, 0xdbd518, 0x0, 0x7fe57b57d470, 0xb)
        /root/.gvm/gos/go1.3.1/src/pkg/net/fd_unix.go:419 +0x343
net.(*TCPListener).AcceptTCP(0xc20808a008, 0xc2081c04e8, 0x0, 0x0)
        /root/.gvm/gos/go1.3.1/src/pkg/net/tcpsock_posix.go:234 +0x5d
net.(*TCPListener).Accept(0xc20808a008, 0x0, 0x0, 0x0, 0x0)
        /root/.gvm/gos/go1.3.1/src/pkg/net/tcpsock_posix.go:244 +0x4b
net/http.(*Server).Serve(0xc2080aa360, 0x7fe57b57f7c0, 0xc20808a008, 0x0, 0x0)
        /root/.gvm/gos/go1.3.1/src/pkg/net/http/server.go:1698 +0x91
net/http.Serve(0x7fe57b57f7c0, 0xc20808a008, 0x7fe57b5852f0, 0xc2081fe100, 0x0, 0x0)
        /root/.gvm/gos/go1.3.1/src/pkg/net/http/server.go:1576 +0x7c
github.com/influxdb/influxdb/admin.(*HttpServer).ListenAndServe(0xc208022d20)
        /root/gocodez/src/github.com/influxdb/influxdb/admin/http_server.go:41 +0x1ad
created by github.com/influxdb/influxdb/server.(*Server).ListenAndServe
        /root/gocodez/src/github.com/influxdb/influxdb/server/server.go:140 +0x52b

goroutine 31 [chan receive, 63 minutes]:
github.com/influxdb/influxdb/server.(*Server).startReportingLoop(0xc208044090, 0xc2081aa260)
        /root/gocodez/src/github.com/influxdb/influxdb/server/server.go:229 +0xc9
created by github.com/influxdb/influxdb/server.(*Server).ListenAndServe
        /root/gocodez/src/github.com/influxdb/influxdb/server/server.go:210 +0x962

goroutine 1635 [syscall]:
github.com/influxdb/rocksdb._Cfunc_rocksdb_compact_range(0x7fe5703dd170, 0x0, 0x0, 0x0, 0x0)
        github.com/influxdb/rocksdb/_obj/_cgo_defun.c:161 +0x31
github.com/influxdb/rocksdb.(*DB).CompactRange(0xc20803aab0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /root/gocodez/src/github.com/influxdb/rocksdb/db.go:311 +0xa9
github.com/influxdb/influxdb/datastore/storage.RocksDB.Compact(0xc20803aab0, 0xc20803aa90, 0xc20803aaa0, 0xc20803aab8, 0xc20803aac0, 0xc20819b470, 0x2e)
        /root/gocodez/src/github.com/influxdb/influxdb/datastore/storage/rocksdb.go:92 +0x56
github.com/influxdb/influxdb/datastore/storage.(*RocksDB).Compact(0xc20819dbc0)
        <autogenerated>:46 +0x91
github.com/influxdb/influxdb/datastore.(*Shard).executeDeleteQuery(0xc2081b6410, 0xc20af337c0, 0x7fe57b591250, 0x0, 0x0, 0x0)
        /root/gocodez/src/github.com/influxdb/influxdb/datastore/shard.go:330 +0x312
github.com/influxdb/influxdb/datastore.(*Shard).Query(0xc2081b6410, 0xc20af337c0, 0x7fe57b591250, 0x0, 0x0, 0x0)
        /root/gocodez/src/github.com/influxdb/influxdb/datastore/shard.go:102 +0x1cf
github.com/influxdb/influxdb/cluster.(*ShardData).Query(0xc208199860, 0xc20af337c0, 0xc209116700)
        /root/gocodez/src/github.com/influxdb/influxdb/cluster/shard.go:327 +0xb0f
created by github.com/influxdb/influxdb/coordinator.(*Coordinator).queryShards
        /root/gocodez/src/github.com/influxdb/influxdb/coordinator/coordinator.go:273 +0x340

goroutine 1592 [IO wait]:
net.runtime_pollWait(0x7fe57b5803f8, 0x72, 0x0)
        /root/.gvm/gos/go1.3.1/src/pkg/runtime/netpoll.goc:146 +0x66
net.(*pollDesc).Wait(0xc20a2560d0, 0x72, 0x0, 0x0)
        /root/.gvm/gos/go1.3.1/src/pkg/net/fd_poll_runtime.go:84 +0x46
net.(*pollDesc).WaitRead(0xc20a2560d0, 0x0, 0x0)
        /root/.gvm/gos/go1.3.1/src/pkg/net/fd_poll_runtime.go:89 +0x42
net.(*netFD).Read(0xc20a256070, 0xc20816c000, 0x1000, 0x1000, 0x0, 0x7fe57b57d470, 0xb)
        /root/.gvm/gos/go1.3.1/src/pkg/net/fd_unix.go:242 +0x34c
net.(*conn).Read(0xc20d234000, 0xc20816c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /root/.gvm/gos/go1.3.1/src/pkg/net/net.go:122 +0xe7
net/http.(*liveSwitchReader).Read(0xc20992e1a8, 0xc20816c000, 0x1000, 0x1000, 0xe94, 0x0, 0x0)
        /root/.gvm/gos/go1.3.1/src/pkg/net/http/server.go:206 +0xaf
io.(*LimitedReader).Read(0xc20a6a0000, 0xc20816c000, 0x1000, 0x1000, 0x1d, 0x0, 0x0)
        /root/.gvm/gos/go1.3.1/src/pkg/io/io.go:399 +0xd0
bufio.(*Reader).fill(0xc2080aa2a0)
        /root/.gvm/gos/go1.3.1/src/pkg/bufio/bufio.go:97 +0x1b3
bufio.(*Reader).ReadSlice(0xc2080aa2a0, 0xc208186a0a, 0x0, 0x0, 0x0, 0x0, 0x0)
        /root/.gvm/gos/go1.3.1/src/pkg/bufio/bufio.go:298 +0x22c
bufio.(*Reader).ReadLine(0xc2080aa2a0, 0x0, 0x0, 0x0, 0x50b400, 0x0, 0x0)
        /root/.gvm/gos/go1.3.1/src/pkg/bufio/bufio.go:326 +0x69
net/textproto.(*Reader).readLineSlice(0xc20d5a2180, 0x0, 0x0, 0x0, 0x0, 0x0)
        /root/.gvm/gos/go1.3.1/src/pkg/net/textproto/reader.go:55 +0x9d
net/textproto.(*Reader).ReadLine(0xc20d5a2180, 0x0, 0x0, 0x0, 0x0)
        /root/.gvm/gos/go1.3.1/src/pkg/net/textproto/reader.go:36 +0x4e
net/http.ReadRequest(0xc2080aa2a0, 0xc20c796270, 0x0, 0x0)
        /root/.gvm/gos/go1.3.1/src/pkg/net/http/request.go:556 +0xc7
net/http.(*conn).readRequest(0xc20992e180, 0x0, 0x0, 0x0)
        /root/.gvm/gos/go1.3.1/src/pkg/net/http/server.go:577 +0x276
net/http.(*conn).serve(0xc20992e180)
        /root/.gvm/gos/go1.3.1/src/pkg/net/http/server.go:1132 +0x61e
created by net/http.(*Server).Serve
        /root/.gvm/gos/go1.3.1/src/pkg/net/http/server.go:1721 +0x313

rax     0x0
rbx     0x7fe537f9f000
rcx     0xffffffffffffffff
rdx     0x6
rdi     0x1040
rsi     0x10a6
rbp     0x103874d
rsp     0x7fe54d7fa878
r8      0xfefefefefefefeff
r9      0xfefefeff092d6364
r10     0x8
r11     0x202
r12     0x1038780
r13     0x103a900
r14     0x7fe5703b67e8
r15     0x7fe5703b67e8
rip     0x333d432625
rflags  0x202
cs      0x33
fs      0x0
gs      0x0
@toddboom
Copy link
Contributor

@apestov We're actually rewriting the data storage layer completely for v0.9.0, so we won't be using RocksDB or LevelDB any longer. We have noticed some strange issues with RocksDB, so we'd recommend using LevelDB for the v0.8.x series in the meantime instead - that should fix the issues you're having.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@toddboom @apestov