runtime: fatal error: free list corrupted in godoc after hours of idle

[thaustad]

MacOS 10.10.5
go version go1.5 darwin/amd64

Possible dup of #11411.

Please note the two comments marked with //.

$ulimit -n 999; godoc -analysis="type" -http=:6060
// lots of normal looking output removed. some "could not import" and other errors.
2015/10/07 21:14:43 Constructing SSA form...
2015/10/07 21:14:49 SSA construction complete
2015/10/07 21:14:49 Package "github.com/go-gl/glfw/v3.0/glfw" has error (*errors.errorString) without position: cgo failed: [go tool cgo -objdir /var/folders/5y/w2d42zkx7rsc43v1pnj080tm0000gn/T/github.com_go-gl_glfw_v3.0_glfw_C258288875 -- -I /var/folders/5y/w2d42zkx7rsc43v1pnj080tm0000gn/T/github.com_go-gl_glfw_v3.0_glfw_C258288875 clipboard.go context.go error.go glfw.go input.go monitor.go native_darwin.go time.go util.go window.go]: exit status 1
2015/10/07 21:14:49 Computing implements relation...
2015/10/07 21:14:50 Extracting type info...
2015/10/07 21:14:57 Visit instructions...
2015/10/07 21:14:59 Visit instructions complete
2015/10/07 21:14:59 Type analysis complete.
// several hours pass
runtime: free list of span 0x6431dd0:
0xc941e5a020 -> 0xc941e5a7c0 -> 0xc941e5a490 -> 0xc941e5a980 -> 0x206567616b636170 (BAD)
fatal error: free list corrupted

runtime stack:
runtime.throw(0x845d60, 0x13)
    /usr/local/go/src/runtime/panic.go:527 +0x90
runtime.mSpan_Sweep(0x6431dd0, 0x1dd00000100, 0xc80006d601)
    /usr/local/go/src/runtime/mgcsweep.go:186 +0x800
runtime.sweepone(0x40c72)
    /usr/local/go/src/runtime/mgcsweep.go:97 +0x154
runtime.gosweepone.func1()
    /usr/local/go/src/runtime/mgcsweep.go:109 +0x21
runtime.systemstack(0xc82001d500)
    /usr/local/go/src/runtime/asm_amd64.s:262 +0x79
runtime.mstart()
    /usr/local/go/src/runtime/proc1.go:674

goroutine 3 [running]:
runtime.systemstack_switch()
    /usr/local/go/src/runtime/asm_amd64.s:216 fp=0xc82002cf58 sp=0xc82002cf50
runtime.gosweepone(0x0)
    /usr/local/go/src/runtime/mgcsweep.go:110 +0x3d fp=0xc82002cf80 sp=0xc82002cf58
runtime.bgsweep(0xc820016070)
    /usr/local/go/src/runtime/mgcsweep.go:54 +0xb6 fp=0xc82002cfb8 sp=0xc82002cf80
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc82002cfc0 sp=0xc82002cfb8
created by runtime.gcenable
    /usr/local/go/src/runtime/mgc.go:205 +0x53

goroutine 1 [IO wait, 191 minutes]:
net.runtime_pollWait(0x1917ca0, 0x72, 0xc8200680a0)
    /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc82373c680, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82373c680, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc82373c620, 0x0, 0x1961ef0, 0xc974f459c0)
    /usr/local/go/src/net/fd_unix.go:408 +0x27c
net.(*TCPListener).AcceptTCP(0xc820072250, 0x5f680, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:254 +0x4d
net/http.tcpKeepAliveListener.Accept(0xc820072250, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2135 +0x41
net/http.(*Server).Serve(0xc8238463c0, 0x1917d60, 0xc820072250, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1887 +0xb3
net/http.(*Server).ListenAndServe(0xc8238463c0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1877 +0x136
net/http.ListenAndServe(0x7fff5fbffc1b, 0x5, 0x1916cc8, 0xc82006cdb0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1967 +0x8f
main.main()
    /var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/gopath/src/golang.org/x/tools/cmd/godoc/main.go:314 +0x18fd

goroutine 17 [syscall, 367 minutes, locked to thread]:
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1

goroutine 1425 [chan receive, 367 minutes]:
golang.org/x/tools/godoc.(*Corpus).refreshMetadataLoop(0xc8201a4000)
    /var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/gopath/src/golang.org/x/tools/godoc/meta.go:140 +0x3a
created by golang.org/x/tools/godoc.(*Corpus).Init
    /var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/gopath/src/golang.org/x/tools/godoc/corpus.go:145 +0x8a

[bradfitz]

/cc @aclements

[ianlancetaylor]

Is it possible for you to try this with 1.5.1?

[thaustad]

This isn't reproducible even on 1.5. It might have happened once more since the release of 1.5, but I don't remember exactly. I have upgraded to 1.5.1 and will let you know if it happens again.

[aclements]

I'm not sure what's going on, but I will note that "0x206567616b636170" is the string "package ".

Do you know if anything in particular was happening around the time when it crashed? It looks like the godoc server hasn't done anything at all for 191 minutes, meaning that this is a background sweep following a periodic GC, but it would be good if we can confirm that.

[thaustad]

godoc was not in use when it crashed. I was away from the computer when it happened, so 191 minutes sounds right.

[helinwang]

I am having the same problem on my http server in golang 1.5.1

untime: free list of span 0x7f470fc0bd90:
0xc820081400 -> 0xc89aa61799 (BAD)
fatal error: free list corrupted

runtime stack:
runtime.throw(0x7f7c30, 0x13)
        /usr/local/go/src/runtime/panic.go:527 +0x90
runtime.mSpan_Sweep(0x7f470fc0bd90, 0x1d00000000, 0x1)
        /usr/local/go/src/runtime/mgcsweep.go:186 +0x800
runtime.sweepone(0x0)
        /usr/local/go/src/runtime/mgcsweep.go:97 +0x154
runtime.gosweepone.func1()
        /usr/local/go/src/runtime/mgcsweep.go:109 +0x21
runtime.systemstack(0x7f46fd7f9dd8)
        /usr/local/go/src/runtime/asm_amd64.s:278 +0xab
runtime.gosweepone(0xb85540)
        /usr/local/go/src/runtime/mgcsweep.go:110 +0x3d
runtime.deductSweepCredit(0x12000, 0x9)
        /usr/local/go/src/runtime/mgcsweep.go:341 +0xc6
runtime.largeAlloc(0x10004, 0x1, 0x7f470fc0e250)
        /usr/local/go/src/runtime/malloc.go:743 +0x7d
runtime.mallocgc.func3()
        /usr/local/go/src/runtime/malloc.go:634 +0x33
runtime.systemstack(0xc8205ecf40)
        /usr/local/go/src/runtime/asm_amd64.s:262 +0x79
runtime.mstart()
        /usr/local/go/src/runtime/proc1.go:674

[ianlancetaylor]

@helinwang Can you easily reproduce this problem? Does your program use the unsafe package? Does it use cgo? Have you tried running it under the race detector? Thanks.

[helinwang]

@ianlancetaylor I am using cgo, easily reproducible in go 1.5 and go 1.5.1. Do you think it could be caused by memory leak in the c part? I doubt if it because of race --- the code is http request handling. The only shared object across thread is http.Client{} (I thought that is thread safe). There is no shared object.

[aclements]

If it is because of C code, it wouldn't be a memory leak in the C code, but rather an incorrect use of cgo corrupting the Go heap. For example, C code manipulating a pointer in the Go heap could cause this.

Can you share your code?

[helinwang]

@aclements I can't share the entire code, the main part is in this line

func foo(data []int16) []byte {
    dst := make([]byte, 1000)
    C.Foo((*C.uchar)(&dst[0]), C.int(len(dst)), (*C.short)(&data[0]), C.int(len(data)))
    return dst
}

I don't think (will double check) the c code will write to data beyond len(dst)
does it look problematic to you besides that?

[ianlancetaylor]

@helinwang That code looks fine assuming there is no buffer overrun and assuming the C code does not try to retain a copy of the Go pointer.

[helinwang]

@ianlancetaylor @aclements thanks for the response, it's indeed a bug in my C code that overrun the c buffer that collapsed the heap memory.