runtime: fatal error: MSpanList_Remove

[googollee]

Please answer these questions before submitting your issue. Thanks!

1. What version of Go are you using (go version)?
   go version go1.6 linux/amd64
2. What operating system and processor architecture are you using (go env)?
   Linux xxxx 3.19.0-25-generic unknown pc's #26~14.04.1-Ubuntu SMP Fri Jul 24 21:16:20 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
3. What did you do?
   If possible, provide a recipe for reproducing the error.
   A complete runnable program is good.
   A link on play.golang.org is best.

Our program is a long TCP connection server, which receiving data from a TCP and forwarding it to other. When it upgrade to 1.6, it run a long time (around 4 days) then panic with same error. Log attached below:

failed MSpanList_Remove 0x7f27485cf7f0 0x7f27481d3aa8 0xb6f170 0xb6f180
fatal error: MSpanList_Remove

runtime stack:
runtime.throw(0x947fd0, 0x10)
        /opt/go1.6/src/runtime/panic.go:530 +0x90
runtime.(*mSpanList).remove(0xb6f180, 0x7f27485cf7f0)
        /opt/go1.6/src/runtime/mheap.go:911 +0x1ad
runtime.(*mcentral).freeSpan(0xb6f160, 0x7f27485cf7f0, 0x16, 0xc8cbdc4080, 0xc8cbdc5b80, 0xc847e62100, 0x100000001)
        /opt/go1.6/src/runtime/mcentral.go:178 +0x131
runtime.(*mspan).sweep(0x7f27485cf7f0, 0x3efd50003ef00, 0xc800021b01)
        /opt/go1.6/src/runtime/mgcsweep.go:319 +0x613
runtime.sweepone(0x4)
        /opt/go1.6/src/runtime/mgcsweep.go:112 +0x23e
runtime.gosweepone.func1()
        /opt/go1.6/src/runtime/mgcsweep.go:124 +0x21
runtime.systemstack(0x7f272ab15db8)
        /opt/go1.6/src/runtime/asm_amd64.s:307 +0xab
runtime.gosweepone(0xb6eb08)
        /opt/go1.6/src/runtime/mgcsweep.go:125 +0x3d
runtime.deductSweepCredit(0x8000, 0x0)
        /opt/go1.6/src/runtime/mgcsweep.go:384 +0xc6
runtime.(*mcentral).cacheSpan(0xb70a50, 0x7f27703fd000)
        /opt/go1.6/src/runtime/mcentral.go:36 +0x56
runtime.(*mcache).refill(0x7f278da73000, 0xc900000042, 0xc867dc1380)
        /opt/go1.6/src/runtime/mcache.go:119 +0xcc
runtime.mallocgc.func2()
        /opt/go1.6/src/runtime/malloc.go:642 +0x2b
runtime.systemstack(0xc82002b500)
        /opt/go1.6/src/runtime/asm_amd64.s:291 +0x79
runtime.mstart()
        /opt/go1.6/src/runtime/proc.go:1048

1. What did you expect to see?
   No crash
2. What did you see instead?
   Crash

[googollee]

There is a old issue with same panic backtrace: #14831

[davecheney]

@googollee #14831 was due to a bug in some code called from cgo, does your code use cgo ?

Is your code available ? Can you provide a sample which reproduces the problem ?

[bits01]

@davecheney to be correct, #14831 did not involve cgo at all, it was just a syscall (ioctl) and there was a bug in a proprietary kernel driver that in some edge case went past the provided buffer by 4 bytes, resulting in memory corruption.
@googollee if you are dealing with memory corruptions, they can happen with syscalls, or use of the unsafe package (pointers) if there's a bug in the pointer math or type conversions via unsafe.Pointer/uintptr

[googollee]

There is no cgo or syscall or unsafe in my code. I will try to make a sample to reproduce it, but the code is really complex and mess ,and panic after a long run. I'm not sure which part cause this issue now.

[davecheney]

The first thing I think your should do is build a race enabled version of your application, go build -race and deploy that.

On 28 Mar 2016, at 19:41, Googol Lee notifications@github.com wrote:

There is no cgo or syscall or unsafe in my code. I will try to make a sample to reproduce it, but the code is really complex and mess ,and panic after a long run. I'm not sure which part cause this issue now.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub

[googollee]

OK. I will try.

[ianlancetaylor]

CC @aclements @RLH

[aclements]

failed MSpanList_Remove 0x7f27485cf7f0 0x7f27481d3aa8 0xb6f170 0xb6f180

Looks like the span was on nonempty, but was being removed from empty because its freelist was nil. Unfortunately, I don't think there's enough information in the traceback to glean which of these conditions was wrong.

[bits01]

@googollee it may be worth looking at the dependencies/3rd party libs you may be using and they may be using in turn. Your code may not be doing anything with the unsafe package/syscall/cgo directly but some dependency may.

[googollee]

I try go build -race and hope to get some log to check now.

@bits01 I have checked all 3rd party libs and all of them are not using cgo/unsafe/sysctl.

[googollee]

I find some race code and try to fix them. After that I will check if the program will panic again.

Thanks.

[bradfitz]

I'm going to close this until there's something actionable for us to do. Feel free to reopen if you have more details to suggest it's a bug in Go.