Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fanout continuous query causes panic #534

Closed
jjmalina opened this issue May 12, 2014 · 3 comments
Closed

fanout continuous query causes panic #534

jjmalina opened this issue May 12, 2014 · 3 comments
Milestone
0.6.3

Comments

@jjmalina
Copy link

I have this continuous fanout query:
SELECT * FROM events INTO events.col1.[col1];

When I write this example data at https://dl.dropboxusercontent.com/u/6854963/fanout.json
curl -H "Content-Type: application/json" -X POST 'http://localhost:8086/db/test/series?u=root&p=root' --data "@fanout.json"

InfluxDB (v0.6.2 (git: 383d79a) (leveldb: 1.15)) logs the following lines

[2014/05/12 13:54:10 EDT] [DEBG] (api/http.(*HttpServer).tryAsDbUserAndClusterAdmin:697) Trying to auth as a db user
[2014/05/12 13:54:10 EDT] [DEBG] (coordinator.(*CoordinatorImpl).AuthenticateDbUser:785) (raft:6d635e7) Authenticating password for test:root
[2014/05/12 13:54:10 EDT] [DEBG] (api/http.(*HttpServer).tryAsDbUserAndClusterAdmin:700) Authenticating as a db user failed with Invalid username/password (401)
[2014/05/12 13:54:10 EDT] [DEBG] (wal.(*WAL).processAppendEntry:306) appending request 9
[2014/05/12 13:54:10 EDT] [DEBG] (wal.(*WAL).processAppendEntry:318) requestsSinceRotation: 9
[2014/05/12 13:54:10 EDT] [DEBG] (wal.(*WAL).conditionalBookmarkAndIndex:517) requestsSinceLastIndex: 9
[2014/05/12 13:54:10 EDT] [DEBG] (wal.(*WAL).flush:532) Fsyncing the log file to disk

and then panics. The points that were sent are written but nothing is written by the continuous query.

Panic stack trace:

panic: runtime error: index out of range

goroutine 20 [running]:
runtime.panic(0x4447300, 0x49cf3b7)
    /usr/local/Cellar/go/1.2.1/libexec/src/pkg/runtime/panic.c:266 +0xb6
datastore.(*LevelDbShard).Write(0xc21011ab00, 0xc2101a58e6, 0x4, 0xc2101ca1e0, 0x0, ...)
    /private/tmp/influxdb-UzZW/influxdb/src/datastore/leveldb_shard.go:86 +0x86a
datastore.(*LevelDbShardDatastore).Write(0xc21007ca80, 0xc2101cb480, 0x0, 0x0)
    /private/tmp/influxdb-UzZW/influxdb/src/datastore/leveldb_shard_datastore.go:167 +0x116
cluster.(*WriteBuffer).write(0xc2100fed90, 0xc2101cb480)
    /private/tmp/influxdb-UzZW/influxdb/src/cluster/write_buffer.go:87 +0x86
cluster.(*WriteBuffer).handleWrites(0xc2100fed90)
    /private/tmp/influxdb-UzZW/influxdb/src/cluster/write_buffer.go:78 +0xb7
created by cluster.NewWriteBuffer
    /private/tmp/influxdb-UzZW/influxdb/src/cluster/write_buffer.go:43 +0x24f

goroutine 1 [IO wait]:
net.runtime_pollWait(0x4bbe208, 0x72, 0x0)
    /usr/local/Cellar/go/1.2.1/libexec/src/pkg/runtime/netpoll.goc:116 +0x6a
net.(*pollDesc).Wait(0xc2101285a0, 0x72, 0x4bbaf60, 0x23)
    /usr/local/Cellar/go/1.2.1/libexec/src/pkg/net/fd_poll_runtime.go:81 +0x34
net.(*pollDesc).WaitRead(0xc2101285a0, 0x23, 0x4bbaf60)
    /usr/local/Cellar/go/1.2.1/libexec/src/pkg/net/fd_poll_runtime.go:86 +0x30
net.(*netFD).accept(0xc210128540, 0x45b7e60, 0x0, 0x4bbaf60, 0x23)
    /usr/local/Cellar/go/1.2.1/libexec/src/pkg/net/fd_unix.go:382 +0x2c2
net.(*TCPListener).AcceptTCP(0xc210132060, 0x418cafb, 0x4e61aa0, 0x418cafb)
    /usr/local/Cellar/go/1.2.1/libexec/src/pkg/net/tcpsock_posix.go:233 +0x47
net.(*TCPListener).Accept(0xc210132060, 0x4bc32b8, 0xc210198bc0, 0xc2101a8700, 0x0)
    /usr/local/Cellar/go/1.2.1/libexec/src/pkg/net/tcpsock_posix.go:243 +0x27
net/http.(*Server).Serve(0xc2100a90a0, 0x4bbd3b8, 0xc210132060, 0x0, 0x0)
    /usr/local/Cellar/go/1.2.1/libexec/src/pkg/net/http/server.go:1622 +0x91
api/http.(*HttpServer).serveListener(0xc2100bb460, 0x4bbd3b8, 0xc210132060, 0xc210132068)
    /private/tmp/influxdb-UzZW/influxdb/src/api/http/api.go:185 +0x98
api/http.(*HttpServer).Serve(0xc2100bb460, 0x4bbd3b8, 0xc210132060)
    /private/tmp/influxdb-UzZW/influxdb/src/api/http/api.go:155 +0xc7c
api/http.(*HttpServer).ListenAndServe(0xc2100bb460)
    /private/tmp/influxdb-UzZW/influxdb/src/api/http/api.go:79 +0x16c
server.(*Server).ListenAndServe(0xc2100e01c0, 0xc2100e01c0, 0x0)
    /private/tmp/influxdb-UzZW/influxdb/src/server/server.go:153 +0x7a6
main.main()
    /private/tmp/influxdb-UzZW/influxdb/src/daemon/influxd.go:155 +0xb97

goroutine 3 [syscall]:
os/signal.loop()
    /usr/local/Cellar/go/1.2.1/libexec/src/pkg/os/signal/signal_unix.go:21 +0x1e
created by os/signal.init·1
    /usr/local/Cellar/go/1.2.1/libexec/src/pkg/os/signal/signal_unix.go:27 +0x31

goroutine 4 [chan receive]:
code.google.com/p/log4go.ConsoleLogWriter.run(0xc21006f000, 0x4bbb1b8, 0xc210000008)
    /private/tmp/influxdb-UzZW/influxdb/src/code.google.com/p/log4go/termlog.go:27 +0x60
created by code.google.com/p/log4go.NewConsoleLogWriter
    /private/tmp/influxdb-UzZW/influxdb/src/code.google.com/p/log4go/termlog.go:19 +0x67

goroutine 5 [select]:
code.google.com/p/log4go.func·002()
    /private/tmp/influxdb-UzZW/influxdb/src/code.google.com/p/log4go/filelog.go:84 +0x84c
created by code.google.com/p/log4go.NewFileLogWriter
    /private/tmp/influxdb-UzZW/influxdb/src/code.google.com/p/log4go/filelog.go:116 +0x2d1

goroutine 6 [syscall]:
runtime.goexit()
    /usr/local/Cellar/go/1.2.1/libexec/src/pkg/runtime/proc.c:1394

goroutine 7 [finalizer wait]:
runtime.park(0x4013860, 0x49d2da0, 0x49d1228)
    /usr/local/Cellar/go/1.2.1/libexec/src/pkg/runtime/proc.c:1342 +0x66
runfinq()
    /usr/local/Cellar/go/1.2.1/libexec/src/pkg/runtime/mgc0.c:2279 +0x84
runtime.goexit()
    /usr/local/Cellar/go/1.2.1/libexec/src/pkg/runtime/proc.c:1394

goroutine 8 [chan receive]:
wal.(*WAL).processEntries(0xc21007ad00)
    /private/tmp/influxdb-UzZW/influxdb/src/wal/wal.go:252 +0x3f
created by wal.NewWAL
    /private/tmp/influxdb-UzZW/influxdb/src/wal/wal.go:103 +0x9f3

goroutine 9 [sleep]:
time.Sleep(0x8bb2c97000)
    /usr/local/Cellar/go/1.2.1/libexec/src/pkg/runtime/time.goc:31 +0x31
cluster.func·001()
    /private/tmp/influxdb-UzZW/influxdb/src/cluster/cluster_configuration.go:133 +0x35
created by cluster.(*ClusterConfiguration).CreateFutureShardsAutomaticallyBeforeTimeComes
    /private/tmp/influxdb-UzZW/influxdb/src/cluster/cluster_configuration.go:138 +0x63

goroutine 11 [chan receive]:
main.waitForSignals(0x4bbd338, 0xc2100e01c0)
    /private/tmp/influxdb-UzZW/influxdb/src/daemon/null_profiler.go:23 +0x126
created by main.startProfiler
    /private/tmp/influxdb-UzZW/influxdb/src/daemon/null_profiler.go:15 +0x38

goroutine 12 [IO wait]:
net.runtime_pollWait(0x4bbe358, 0x72, 0x0)
    /usr/local/Cellar/go/1.2.1/libexec/src/pkg/runtime/netpoll.goc:116 +0x6a
net.(*pollDesc).Wait(0xc2100e0290, 0x72, 0x4bbaf60, 0x23)
    /usr/local/Cellar/go/1.2.1/libexec/src/pkg/net/fd_poll_runtime.go:81 +0x34
net.(*pollDesc).WaitRead(0xc2100e0290, 0x23, 0x4bbaf60)
    /usr/local/Cellar/go/1.2.1/libexec/src/pkg/net/fd_poll_runtime.go:86 +0x30
net.(*netFD).accept(0xc2100e0230, 0x45b7e60, 0x0, 0x4bbaf60, 0x23)
    /usr/local/Cellar/go/1.2.1/libexec/src/pkg/net/fd_unix.go:382 +0x2c2
net.(*TCPListener).AcceptTCP(0xc2100bc678, 0x18, 0xc2100d6810, 0x418c8d3)
    /usr/local/Cellar/go/1.2.1/libexec/src/pkg/net/tcpsock_posix.go:233 +0x47
net.(*TCPListener).Accept(0xc2100bc678, 0x0, 0x0, 0x0, 0x0)
    /usr/local/Cellar/go/1.2.1/libexec/src/pkg/net/tcpsock_posix.go:243 +0x27
net/http.(*Server).Serve(0xc2100b8640, 0x4bbd3b8, 0xc2100bc678, 0x0, 0x0)
    /usr/local/Cellar/go/1.2.1/libexec/src/pkg/net/http/server.go:1622 +0x91
coordinator.func·007()
    /private/tmp/influxdb-UzZW/influxdb/src/coordinator/raft_server.go:530 +0x3a
created by coordinator.(*RaftServer).Serve
    /private/tmp/influxdb-UzZW/influxdb/src/coordinator/raft_server.go:534 +0x4d9

goroutine 17 [select]:
coordinator.(*RaftServer).raftLeaderLoop(0xc2100d7b00, 0xc21011a680)
    /private/tmp/influxdb-UzZW/influxdb/src/coordinator/raft_server.go:430 +0x29c
created by coordinator.(*RaftServer).raftEventHandler
    /private/tmp/influxdb-UzZW/influxdb/src/coordinator/raft_server.go:419 +0x1d0

goroutine 14 [select]:
github.com/goraft/raft.(*server).leaderLoop(0xc2100b4a20)
    /private/tmp/influxdb-UzZW/influxdb/src/github.com/goraft/raft/server.go:765 +0x5fe
github.com/goraft/raft.(*server).loop(0xc2100b4a20)
    /private/tmp/influxdb-UzZW/influxdb/src/github.com/goraft/raft/server.go:568 +0x33f
created by github.com/goraft/raft.(*server).Start
    /private/tmp/influxdb-UzZW/influxdb/src/github.com/goraft/raft/server.go:472 +0x7af

goroutine 15 [select]:
coordinator.(*RaftServer).CompactLog(0xc2100d7b00)
    /private/tmp/influxdb-UzZW/influxdb/src/coordinator/raft_server.go:320 +0x2ef
created by coordinator.(*RaftServer).startRaft
    /private/tmp/influxdb-UzZW/influxdb/src/coordinator/raft_server.go:374 +0x375

goroutine 19 [IO wait]:
net.runtime_pollWait(0x4bbe2b0, 0x72, 0x0)
    /usr/local/Cellar/go/1.2.1/libexec/src/pkg/runtime/netpoll.goc:116 +0x6a
net.(*pollDesc).Wait(0xc210128290, 0x72, 0x4bbaf60, 0x23)
    /usr/local/Cellar/go/1.2.1/libexec/src/pkg/net/fd_poll_runtime.go:81 +0x34
net.(*pollDesc).WaitRead(0xc210128290, 0x23, 0x4bbaf60)
    /usr/local/Cellar/go/1.2.1/libexec/src/pkg/net/fd_poll_runtime.go:86 +0x30
net.(*netFD).accept(0xc210128230, 0x45b7e60, 0x0, 0x4bbaf60, 0x23)
    /usr/local/Cellar/go/1.2.1/libexec/src/pkg/net/fd_unix.go:382 +0x2c2
net.(*TCPListener).AcceptTCP(0xc210132038, 0xc21012b1c0, 0x0, 0x4bbd388)
    /usr/local/Cellar/go/1.2.1/libexec/src/pkg/net/tcpsock_posix.go:233 +0x47
net.(*TCPListener).Accept(0xc210132038, 0xc21012b1c0, 0x4fa1f38, 0x1, 0x1)
    /usr/local/Cellar/go/1.2.1/libexec/src/pkg/net/tcpsock_posix.go:243 +0x27
coordinator.(*ProtobufServer).ListenAndServe(0xc2100b7c00)
    /private/tmp/influxdb-UzZW/influxdb/src/coordinator/protobuf_server.go:64 +0x1c7
created by server.(*Server).ListenAndServe
    /private/tmp/influxdb-UzZW/influxdb/src/server/server.go:116 +0x218

goroutine 23 [IO wait]:
net.runtime_pollWait(0x4bbe160, 0x72, 0x0)
    /usr/local/Cellar/go/1.2.1/libexec/src/pkg/runtime/netpoll.goc:116 +0x6a
net.(*pollDesc).Wait(0xc21011b990, 0x72, 0x4bbaf60, 0x23)
    /usr/local/Cellar/go/1.2.1/libexec/src/pkg/net/fd_poll_runtime.go:81 +0x34
net.(*pollDesc).WaitRead(0xc21011b990, 0x23, 0x4bbaf60)
    /usr/local/Cellar/go/1.2.1/libexec/src/pkg/net/fd_poll_runtime.go:86 +0x30
net.(*netFD).accept(0xc21011b930, 0x45b7e60, 0x0, 0x4bbaf60, 0x23)
    /usr/local/Cellar/go/1.2.1/libexec/src/pkg/net/fd_unix.go:382 +0x2c2
net.(*TCPListener).AcceptTCP(0xc2100000a0, 0x18, 0xc210159010, 0x418c8d3)
    /usr/local/Cellar/go/1.2.1/libexec/src/pkg/net/tcpsock_posix.go:233 +0x47
net.(*TCPListener).Accept(0xc2100000a0, 0x50, 0x49ddb00, 0x18, 0x0)
    /usr/local/Cellar/go/1.2.1/libexec/src/pkg/net/tcpsock_posix.go:243 +0x27
net/http.(*Server).Serve(0xc2100be0a0, 0x4bbd3b8, 0xc2100000a0, 0x0, 0x0)
    /usr/local/Cellar/go/1.2.1/libexec/src/pkg/net/http/server.go:1622 +0x91
net/http.Serve(0x4bbd3b8, 0xc2100000a0, 0x4bc3268, 0xc2100b3340, 0x23, ...)
    /usr/local/Cellar/go/1.2.1/libexec/src/pkg/net/http/server.go:1561 +0x70
admin.(*HttpServer).ListenAndServe(0xc2100b7c80)
    /private/tmp/influxdb-UzZW/influxdb/src/admin/http_server.go:35 +0x170
created by server.(*Server).ListenAndServe
    /private/tmp/influxdb-UzZW/influxdb/src/server/server.go:130 +0x460

goroutine 36 [runnable]:
unicode/utf8.DecodeRuneInString(0xc2101a5260, 0x12, 0xc, 0xc2101d9f80)
    /usr/local/Cellar/go/1.2.1/libexec/src/pkg/unicode/utf8/utf8.go:229
regexp.(*Regexp).replaceAll(0xc2100ffbe0, 0x0, 0x0, 0x0, 0xc2101a5260, ...)
    /usr/local/Cellar/go/1.2.1/libexec/src/pkg/regexp/regexp.go:498 +0x6db
regexp.(*Regexp).ReplaceAllStringFunc(0xc2100ffbe0, 0xc2101a5260, 0x12, 0x4fe16b0, 0xc21017c8b0, ...)
    /usr/local/Cellar/go/1.2.1/libexec/src/pkg/regexp/regexp.go:457 +0xa0
coordinator.(*CoordinatorImpl).InterpolateValuesAndCommit(0xc2100b9dc0, 0xc2101d73f0, 0x2c, 0xc2101a58e6, 0x4, ...)
    /private/tmp/influxdb-UzZW/influxdb/src/coordinator/coordinator.go:554 +0x6b5
coordinator.(*CoordinatorImpl).ProcessContinuousQueries(0xc2100b9dc0, 0xc2101a58e6, 0x4, 0xc2101ca0f0)
    /private/tmp/influxdb-UzZW/influxdb/src/coordinator/coordinator.go:511 +0x313
coordinator.(*CoordinatorImpl).WriteSeriesData(0xc2100b9dc0, 0x4bc34a8, 0xc21011a740, 0xc2101a58e6, 0x4, ...)
    /private/tmp/influxdb-UzZW/influxdb/src/coordinator/coordinator.go:484 +0x1f1
api/http.func·005(0x4bc34a8, 0xc21011a740, 0xc2101a58ed, 0x4, 0xc2101a58f4)
    /private/tmp/influxdb-UzZW/influxdb/src/api/http/api.go:366 +0x334
api/http.yieldUser(0x4bc34a8, 0xc21011a740, 0x4fe1c20, 0xc2101a58f4, 0x4, ...)
    /private/tmp/influxdb-UzZW/influxdb/src/api/http/api.go:473 +0x3b
api/http.(*HttpServer).tryAsClusterAdmin(0xc2100bb460, 0x4bc3410, 0xc21012cb40, 0xc21016aea0, 0x4fe1c20)
    /private/tmp/influxdb-UzZW/influxdb/src/api/http/api.go:535 +0x34a
api/http.(*HttpServer).tryAsDbUserAndClusterAdmin(0xc2100bb460, 0x4bc3410, 0xc21012cb40, 0xc21016aea0, 0x4fe1c20)
    /private/tmp/influxdb-UzZW/influxdb/src/api/http/api.go:705 +0x268
api/http.(*HttpServer).writePoints(0xc2100bb460, 0x4bc3410, 0xc21012cb40, 0xc21016aea0)
    /private/tmp/influxdb-UzZW/influxdb/src/api/http/api.go:373 +0x1e5
api/http.*HttpServer.(api/http.writePoints)·fm(0x4bc3410, 0xc21012cb40, 0xc21016aea0)
    /private/tmp/influxdb-UzZW/influxdb/src/api/http/api.go:105 +0x44
api/http.func·031(0x4bc3410, 0xc21012cb40, 0xc21016aea0)
    /private/tmp/influxdb-UzZW/influxdb/src/api/http/cors.go:13 +0x180
net/http.HandlerFunc.ServeHTTP(0xc21012b300, 0x4bc3410, 0xc21012cb40, 0xc21016aea0)
    /usr/local/Cellar/go/1.2.1/libexec/src/pkg/net/http/server.go:1220 +0x40
github.com/bmizerany/pat.(*PatternServeMux).ServeHTTP(0xc210132068, 0x4bc3410, 0xc21012cb40, 0xc21016aea0)
    /private/tmp/influxdb-UzZW/influxdb/src/github.com/bmizerany/pat/mux.go:109 +0x1be
net/http.serverHandler.ServeHTTP(0xc2100a90a0, 0x4bc3410, 0xc21012cb40, 0xc21016aea0)
    /usr/local/Cellar/go/1.2.1/libexec/src/pkg/net/http/server.go:1597 +0x16e
net/http.(*conn).serve(0xc2101a8700)
    /usr/local/Cellar/go/1.2.1/libexec/src/pkg/net/http/server.go:1167 +0x7b7
created by net/http.(*Server).Serve
    /usr/local/Cellar/go/1.2.1/libexec/src/pkg/net/http/server.go:1644 +0x28b
@jvshahid jvshahid added this to the 0.6.2 milestone May 12, 2014
@toddboom
Copy link
Contributor

@jjmalina does the server start back up afterwards?

@jjmalina
Copy link
Author

@toddboom yeah, if I tell the server to start up it comes back fine. just sending that data with that continuous query present crashes it.

@jvshahid jvshahid modified the milestones: Next release, 0.6.2 May 12, 2014
@toddboom
Copy link
Contributor

@jjmaline thanks. checking it out over here with your dataset. i'll follow up shortly.

@toddboom toddboom modified the milestones: 0.6.3, Next release May 12, 2014
@toddboom toddboom self-assigned this May 12, 2014
@jvshahid jvshahid assigned jvshahid and unassigned toddboom May 13, 2014
jvshahid added a commit that referenced this issue May 27, 2014
@jvshahid @pauldix
Fix #534. Create a new series when interpolating
cf7a846
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
0.6.3
Development

No branches or pull requests
3 participants
@toddboom @jjmalina @jvshahid