Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[0.9.4.2] - Crash / Panic : Span weird #4655

Closed
JulienChampseix opened this issue Nov 4, 2015 · 8 comments
Closed

[0.9.4.2] - Crash / Panic : Span weird #4655

JulienChampseix opened this issue Nov 4, 2015 · 8 comments
Labels
panic

Comments

@JulienChampseix
Copy link

Hi Team,

Version infos :
InfluxDB v0.9.4.2 (git: 0.9.4 da1b59e)

Server 0.9.4.2 crashed suddently, stack log crash:

runtime:objectstart Span weird: p=[http] 2015/11/04 02:00:58 10.13.1.9 - brain [04/Nov/2015:02:00:58 +0000] POST /write?db=brain.applications&p=%5BREDACTED%5D&u=brain HTTP/1.1 204 0 - - e42b53cb-8297-11e5-8f7e-000000000000 615.073µs
0xc900000000 k=0x6480000 s.start=0xc900000000 s.limit=0xc900005a00 s.state=2
fatal error: objectstart: bad pointer in unexpected span

goroutine 54 [running]:
runtime.throw(0xcb2580, 0x2b)
    /home/philip/.gvm/gos/go1.5/src/runtime/panic.go:527 +0x90 fp=0xc820596610 sp=0xc8205965f8
runtime.heapBitsForObject(0xc900000000, 0x0, 0x0, 0xc800000000, 0x7fd4cdcacb90)
    /home/philip/.gvm/gos/go1.5/src/runtime/mbitmap.go:217 +0x287 fp=0xc820596648 sp=0xc820596610
runtime.scanobject(0xc96578bd70, 0xc82002dc20)
    /home/philip/.gvm/gos/go1.5/src/runtime/mgcmark.go:863 +0x239 fp=0xc820596718 sp=0xc820596648
runtime.gcDrainUntilPreempt(0xc82002dc20, 0x7d0)
    /home/philip/.gvm/gos/go1.5/src/runtime/mgcmark.go:726 +0x152 fp=0xc820596750 sp=0xc820596718
runtime.gcBgMarkWorker(0xc82002ca00)
    /home/philip/.gvm/gos/go1.5/src/runtime/mgc.go:1328 +0x474 fp=0xc8205967b8 sp=0xc820596750
runtime.goexit()
    /home/philip/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8205967c0 sp=0xc8205967b8
created by runtime.gcBgMarkStartWorkers
    /home/philip/.gvm/gos/go1.5/src/runtime/mgc.go:1238 +0x93

goroutine 1 [chan receive, 8016 minutes]:
main.(*Main).Run(0xc8ab433f10, 0xc82000a0b0, 0x4, 0x4, 0x0, 0x0)
    /tmp/build/src/github.com/influxdb/influxdb/cmd/influxd/main.go:91 +0xa16
main.main()
    /tmp/build/src/github.com/influxdb/influxdb/cmd/influxd/main.go:42 +0x29b

goroutine 17 [syscall, 8017 minutes, locked to thread]:
runtime.goexit()
    /home/philip/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1

goroutine 5 [syscall, 8017 minutes]:
os/signal.loop()
    /home/philip/.gvm/gos/go1.5/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
    /home/philip/.gvm/gos/go1.5/src/os/signal/signal_unix.go:28 +0x37

goroutine 19 [IO wait, 8017 minutes]:
net.runtime_pollWait(0x7fd4f4a2c380, 0x72, 0xc8200101a0)
    /home/philip/.gvm/gos/go1.5/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc8201204c0, 0x72, 0x0, 0x0)
    /home/philip/.gvm/gos/go1.5/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8201204c0, 0x0, 0x0)
    /home/philip/.gvm/gos/go1.5/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc820120460, 0x0, 0x7fd4f6271788, 0xc820984000)
    /home/philip/.gvm/gos/go1.5/src/net/fd_unix.go:408 +0x27c
net.(*TCPListener).AcceptTCP(0xc82010e0a8, 0xc8201c6ec8, 0x0, 0x0)
    /home/philip/.gvm/gos/go1.5/src/net/tcpsock_posix.go:254 +0x4d
net.(*TCPListener).Accept(0xc82010e0a8, 0x0, 0x0, 0x0, 0x0)
    /home/philip/.gvm/gos/go1.5/src/net/tcpsock_posix.go:264 +0x3d
github.com/influxdb/influxdb/tcp.(*Mux).Serve(0xc82010b3c0, 0x7fd4f4a2b400, 0xc82010e0a8, 0x0, 0x0)
    /tmp/build/src/github.com/influxdb/influxdb/tcp/mux.go:48 +0x4b
created by github.com/influxdb/influxdb/cmd/influxd/run.(*Server).Open.func1
    /tmp/build/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:346 +0xae9

goroutine 20 [chan receive, 8017 minutes]:
github.com/influxdb/influxdb/tcp.(*listener).Accept(0xc82010e0b0, 0x0, 0x0, 0x0, 0x0)
    /tmp/build/src/github.com/influxdb/influxdb/tcp/mux.go:129 +0x6f
github.com/influxdb/influxdb/meta.(*raftLayer).Accept(0xc820188540, 0x0, 0x0, 0x0, 0x0)
    /tmp/build/src/github.com/influxdb/influxdb/meta/store.go:2059 +0x51
github.com/hashicorp/raft.(*NetworkTransport).listen(0xc8201204d0)
    /tmp/build/src/github.com/hashicorp/raft/net_transport.go:346 +0x79
created by github.com/hashicorp/raft.NewNetworkTransport
    /tmp/build/src/github.com/hashicorp/raft/net_transport.go:138 +0x284

goroutine 21 [select, 2 minutes]:
github.com/hashicorp/raft.(*Raft).leaderLoop(0xc82019a000)
    /tmp/build/src/github.com/hashicorp/raft/raft.go:849 +0xcce
github.com/hashicorp/raft.(*Raft).runLeader(0xc82019a000)
    /tmp/build/src/github.com/hashicorp/raft/raft.go:821 +0x936
github.com/hashicorp/raft.(*Raft).run(0xc82019a000)
    /tmp/build/src/github.com/hashicorp/raft/raft.go:602 +0xbb
github.com/hashicorp/raft.(*Raft).(github.com/hashicorp/raft.run)-fm()
    /tmp/build/src/github.com/hashicorp/raft/raft.go:252 +0x20
github.com/hashicorp/raft.(*raftState).goFunc.func1(0xc82019a000, 0xc82018c540)
    /tmp/build/src/github.com/hashicorp/raft/state.go:152 +0x44
created by github.com/hashicorp/raft.(*raftState).goFunc
    /tmp/build/src/github.com/hashicorp/raft/state.go:153 +0x4d

goroutine 22 [select, 147 minutes]:
github.com/hashicorp/raft.(*Raft).runFSM(0xc82019a000)
    /tmp/build/src/github.com/hashicorp/raft/raft.go:509 +0x10eb
github.com/hashicorp/raft.(*Raft).(github.com/hashicorp/raft.runFSM)-fm()
    /tmp/build/src/github.com/hashicorp/raft/raft.go:253 +0x20
github.com/hashicorp/raft.(*raftState).goFunc.func1(0xc82019a000, 0xc82018c550)
    /tmp/build/src/github.com/hashicorp/raft/state.go:152 +0x44
created by github.com/hashicorp/raft.(*raftState).goFunc
    /tmp/build/src/github.com/hashicorp/raft/state.go:153 +0x4d

goroutine 23 [select, 2 minutes]:
github.com/hashicorp/raft.(*Raft).runSnapshots(0xc82019a000)
    /tmp/build/src/github.com/hashicorp/raft/raft.go:1636 +0x493
github.com/hashicorp/raft.(*Raft).(github.com/hashicorp/raft.runSnapshots)-fm()
    /tmp/build/src/github.com/hashicorp/raft/raft.go:254 +0x20
github.com/hashicorp/raft.(*raftState).goFunc.func1(0xc82019a000, 0xc82018c560)
    /tmp/build/src/github.com/hashicorp/raft/state.go:152 +0x44
created by github.com/hashicorp/raft.(*raftState).goFunc
    /tmp/build/src/github.com/hashicorp/raft/state.go:153 +0x4d

goroutine 24 [select, 8017 minutes]:
github.com/influxdb/influxdb/meta.(*localRaft).logLeaderChanges(0xc820107130)
    /tmp/build/src/github.com/influxdb/influxdb/meta/state.go:187 +0x4f5
created by github.com/influxdb/influxdb/meta.(*localRaft).open
    /tmp/build/src/github.com/influxdb/influxdb/meta/state.go:177 +0x1344

goroutine 25 [chan receive, 8017 minutes]:
github.com/influxdb/influxdb/tcp.(*listener).Accept(0xc82010e0b8, 0x0, 0x0, 0x0, 0x0)
    /tmp/build/src/github.com/influxdb/influxdb/tcp/mux.go:129 +0x6f
github.com/influxdb/influxdb/meta.(*Store).serveExecListener(0xc820124280)
    /tmp/build/src/github.com/influxdb/influxdb/meta/store.go:622 +0xa2
created by github.com/influxdb/influxdb/meta.(*Store).Open
    /tmp/build/src/github.com/influxdb/influxdb/meta/store.go:239 +0x364

goroutine 26 [chan receive, 8017 minutes]:
github.com/influxdb/influxdb/tcp.(*listener).Accept(0xc82010e0c0, 0x0, 0x0, 0x0, 0x0)
    /tmp/build/src/github.com/influxdb/influxdb/tcp/mux.go:129 +0x6f
github.com/influxdb/influxdb/meta.(*Store).serveRPCListener(0xc820124280)
    /tmp/build/src/github.com/influxdb/influxdb/meta/store.go:726 +0xa2
created by github.com/influxdb/influxdb/meta.(*Store).Open
    /tmp/build/src/github.com/influxdb/influxdb/meta/store.go:242 +0x3b6

goroutine 30 [select, 2 minutes]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc870602000, 0xc8488c81e0)
    /tmp/build/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:604 +0x567
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /tmp/build/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:240 +0x627

goroutine 39 [select, 8017 minutes]:
github.com/influxdb/influxdb/cmd/influxd/run.(*Server).monitorErrorChan(0xc82011db00, 0xc8201166c0)
    /tmp/build/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:492 +0x145
created by github.com/influxdb/influxdb/cmd/influxd/run.(*Server).Open.func1
    /tmp/build/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:352 +0xc67

goroutine 68 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc86d7ca2a0, 0xc867b8c060)
    /tmp/build/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:604 +0x567
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /tmp/build/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:240 +0x627

goroutine 16 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc8201268c0, 0xc821512660)
    /tmp/build/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:604 +0x567
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /tmp/build/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:240 +0x627
@beckettsean
Copy link
Contributor

@JulienChampseix any recent changes in load on the system? Any possibly correlative system changes?

@beckettsean
Copy link
Contributor

A quick search turns up these Golang issues with the same stack trace: golang/go#12157, golang/go#12158.

@JulienChampseix what OS and CPU architecture are you running?

@otoolep
Copy link
Contributor

otoolep commented Nov 4, 2015

This is a known issue with Go 1.5.1. We are hoping it will be fixed in Go 1.5.2.

#4554

@otoolep otoolep closed this as completed Nov 4, 2015
@otoolep otoolep reopened this Nov 4, 2015
@otoolep
Copy link
Contributor

otoolep commented Nov 4, 2015

Actually, this might be a different issue. I'm going to leave this open for now so we can see if it clears up after 1.5.2 is released and we start building with that.

@JulienChampseix
Copy link
Author

Currently running under : Ubuntu LTS 14.04
CPU : Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz 12Core

thanks @otoolep & @beckettsean for your highlight.

@jwilder jwilder added the panic label Nov 5, 2015
@ssh3
Copy link

ssh3 commented Nov 8, 2015

Hello
The same problem.
OS: Ubuntu 14.04.3 LTS.
CPU: Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz, 20 Core

Thanks for your time.

@JulienChampseix
Copy link
Author

@otoolep 1.5.2 released regarding the go issue about it

@JulienChampseix JulienChampseix mentioned this issue Nov 26, 2015
Closed
@nathanielc
Copy link
Contributor

We have switched to go 1.5.2 closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
panic
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@jwilder @otoolep @JulienChampseix @nathanielc @ssh3 @beckettsean