runtime: fatal error: free list corrupted in godoc after hours of idle #12879

thaustad · 2015-10-08T14:01:38Z

MacOS 10.10.5
go version go1.5 darwin/amd64

Possible dup of #11411.

Please note the two comments marked with //.

$ulimit -n 999; godoc -analysis="type" -http=:6060
// lots of normal looking output removed. some "could not import" and other errors.
2015/10/07 21:14:43 Constructing SSA form...
2015/10/07 21:14:49 SSA construction complete
2015/10/07 21:14:49 Package "github.com/go-gl/glfw/v3.0/glfw" has error (*errors.errorString) without position: cgo failed: [go tool cgo -objdir /var/folders/5y/w2d42zkx7rsc43v1pnj080tm0000gn/T/github.com_go-gl_glfw_v3.0_glfw_C258288875 -- -I /var/folders/5y/w2d42zkx7rsc43v1pnj080tm0000gn/T/github.com_go-gl_glfw_v3.0_glfw_C258288875 clipboard.go context.go error.go glfw.go input.go monitor.go native_darwin.go time.go util.go window.go]: exit status 1
2015/10/07 21:14:49 Computing implements relation...
2015/10/07 21:14:50 Extracting type info...
2015/10/07 21:14:57 Visit instructions...
2015/10/07 21:14:59 Visit instructions complete
2015/10/07 21:14:59 Type analysis complete.
// several hours pass
runtime: free list of span 0x6431dd0:
0xc941e5a020 -> 0xc941e5a7c0 -> 0xc941e5a490 -> 0xc941e5a980 -> 0x206567616b636170 (BAD)
fatal error: free list corrupted

runtime stack:
runtime.throw(0x845d60, 0x13)
    /usr/local/go/src/runtime/panic.go:527 +0x90
runtime.mSpan_Sweep(0x6431dd0, 0x1dd00000100, 0xc80006d601)
    /usr/local/go/src/runtime/mgcsweep.go:186 +0x800
runtime.sweepone(0x40c72)
    /usr/local/go/src/runtime/mgcsweep.go:97 +0x154
runtime.gosweepone.func1()
    /usr/local/go/src/runtime/mgcsweep.go:109 +0x21
runtime.systemstack(0xc82001d500)
    /usr/local/go/src/runtime/asm_amd64.s:262 +0x79
runtime.mstart()
    /usr/local/go/src/runtime/proc1.go:674

goroutine 3 [running]:
runtime.systemstack_switch()
    /usr/local/go/src/runtime/asm_amd64.s:216 fp=0xc82002cf58 sp=0xc82002cf50
runtime.gosweepone(0x0)
    /usr/local/go/src/runtime/mgcsweep.go:110 +0x3d fp=0xc82002cf80 sp=0xc82002cf58
runtime.bgsweep(0xc820016070)
    /usr/local/go/src/runtime/mgcsweep.go:54 +0xb6 fp=0xc82002cfb8 sp=0xc82002cf80
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc82002cfc0 sp=0xc82002cfb8
created by runtime.gcenable
    /usr/local/go/src/runtime/mgc.go:205 +0x53

goroutine 1 [IO wait, 191 minutes]:
net.runtime_pollWait(0x1917ca0, 0x72, 0xc8200680a0)
    /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc82373c680, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82373c680, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc82373c620, 0x0, 0x1961ef0, 0xc974f459c0)
    /usr/local/go/src/net/fd_unix.go:408 +0x27c
net.(*TCPListener).AcceptTCP(0xc820072250, 0x5f680, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:254 +0x4d
net/http.tcpKeepAliveListener.Accept(0xc820072250, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2135 +0x41
net/http.(*Server).Serve(0xc8238463c0, 0x1917d60, 0xc820072250, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1887 +0xb3
net/http.(*Server).ListenAndServe(0xc8238463c0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1877 +0x136
net/http.ListenAndServe(0x7fff5fbffc1b, 0x5, 0x1916cc8, 0xc82006cdb0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1967 +0x8f
main.main()
    /var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/gopath/src/golang.org/x/tools/cmd/godoc/main.go:314 +0x18fd

goroutine 17 [syscall, 367 minutes, locked to thread]:
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1

goroutine 1425 [chan receive, 367 minutes]:
golang.org/x/tools/godoc.(*Corpus).refreshMetadataLoop(0xc8201a4000)
    /var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/gopath/src/golang.org/x/tools/godoc/meta.go:140 +0x3a
created by golang.org/x/tools/godoc.(*Corpus).Init
    /var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/gopath/src/golang.org/x/tools/godoc/corpus.go:145 +0x8a

The text was updated successfully, but these errors were encountered:

bradfitz · 2015-10-08T14:47:01Z

/cc @aclements

ianlancetaylor · 2015-10-08T15:47:52Z

Is it possible for you to try this with 1.5.1?

thaustad · 2015-10-08T18:23:22Z

This isn't reproducible even on 1.5. It might have happened once more since the release of 1.5, but I don't remember exactly. I have upgraded to 1.5.1 and will let you know if it happens again.

aclements · 2015-10-09T18:04:24Z

I'm not sure what's going on, but I will note that "0x206567616b636170" is the string "package ".

Do you know if anything in particular was happening around the time when it crashed? It looks like the godoc server hasn't done anything at all for 191 minutes, meaning that this is a background sweep following a periodic GC, but it would be good if we can confirm that.

thaustad · 2015-10-10T16:55:52Z

godoc was not in use when it crashed. I was away from the computer when it happened, so 191 minutes sounds right.

helinwang · 2015-11-03T00:04:42Z

I am having the same problem on my http server in golang 1.5.1

untime: free list of span 0x7f470fc0bd90:
0xc820081400 -> 0xc89aa61799 (BAD)
fatal error: free list corrupted

runtime stack:
runtime.throw(0x7f7c30, 0x13)
        /usr/local/go/src/runtime/panic.go:527 +0x90
runtime.mSpan_Sweep(0x7f470fc0bd90, 0x1d00000000, 0x1)
        /usr/local/go/src/runtime/mgcsweep.go:186 +0x800
runtime.sweepone(0x0)
        /usr/local/go/src/runtime/mgcsweep.go:97 +0x154
runtime.gosweepone.func1()
        /usr/local/go/src/runtime/mgcsweep.go:109 +0x21
runtime.systemstack(0x7f46fd7f9dd8)
        /usr/local/go/src/runtime/asm_amd64.s:278 +0xab
runtime.gosweepone(0xb85540)
        /usr/local/go/src/runtime/mgcsweep.go:110 +0x3d
runtime.deductSweepCredit(0x12000, 0x9)
        /usr/local/go/src/runtime/mgcsweep.go:341 +0xc6
runtime.largeAlloc(0x10004, 0x1, 0x7f470fc0e250)
        /usr/local/go/src/runtime/malloc.go:743 +0x7d
runtime.mallocgc.func3()
        /usr/local/go/src/runtime/malloc.go:634 +0x33
runtime.systemstack(0xc8205ecf40)
        /usr/local/go/src/runtime/asm_amd64.s:262 +0x79
runtime.mstart()
        /usr/local/go/src/runtime/proc1.go:674

ianlancetaylor · 2015-11-03T00:12:03Z

@helinwang Can you easily reproduce this problem? Does your program use the unsafe package? Does it use cgo? Have you tried running it under the race detector? Thanks.

helinwang · 2015-11-03T00:21:13Z

@ianlancetaylor I am using cgo, easily reproducible in go 1.5 and go 1.5.1. Do you think it could be caused by memory leak in the c part? I doubt if it because of race --- the code is http request handling. ~~The only shared object across thread is http.Client{} (I thought that is thread safe).~~ There is no shared object.

aclements · 2015-11-03T00:42:35Z

If it is because of C code, it wouldn't be a memory leak in the C code, but rather an incorrect use of cgo corrupting the Go heap. For example, C code manipulating a pointer in the Go heap could cause this.

Can you share your code?

helinwang · 2015-11-03T01:05:44Z

@aclements I can't share the entire code, the main part is in this line

func foo(data []int16) []byte {
    dst := make([]byte, 1000)
    C.Foo((*C.uchar)(&dst[0]), C.int(len(dst)), (*C.short)(&data[0]), C.int(len(data)))
    return dst
}

I don't think (will double check) the c code will write to data beyond len(dst)
does it look problematic to you besides that?

ianlancetaylor · 2015-11-03T04:03:31Z

@helinwang That code looks fine assuming there is no buffer overrun and assuming the C code does not try to retain a copy of the Go pointer.

helinwang · 2015-11-03T20:19:30Z

@ianlancetaylor @aclements thanks for the response, it's indeed a bug in my C code that overrun the c buffer that collapsed the heap memory.

ianlancetaylor · 2015-11-03T23:50:23Z

@helinwang Thanks for the follow up.

This bug is still open for the original report by @thaustad.

thaustad · 2015-11-10T23:05:23Z

Got another one. Go 1.5.1 this time, with the same OS.

The pointer decodes to ".com/p/a" which must be part of one of the "code.google.com/p/appengine-go" package names.

I have added date to the command line to see when it panics.

$ulimit -n 999; godoc -analysis="type" -http=:6060; date
// similar output as last time removed
2015/11/09 15:45:16 Type analysis complete.
runtime: free list of span 0x6e2bd40:
0xc944610000 -> 0xc944610070 -> 0xc9446100e0 -> 0xc944610150 -> 0xc944610230 -> 0xc944610310 -> 0xc9446103f0 -> 0xc944610460 -> 0xc9446104d0 -> 0xc944610540 -> 0xc9446105b0 -> 0xc944610620 -> 0xc944610690 -> 0xc944610700 -> 0xc9446107e0 -> 0xc9446108c0 -> 0xc9446109a0 -> 0xc944610a10 -> 0xc944610a80 -> 0xc944610af0 -> 0xc944610b60 -> 0xc944610bd0 -> 0xc944610c40 -> 0xc944610d20 -> 0xc944610e00 -> 0xc944610ee0 -> 0xc944610fc0 -> 0xc9446110a0 -> 0x612f702f6d6f632e (BAD)
fatal error: free list corrupted

runtime stack:
runtime.throw(0x845f20, 0x13)
    /usr/local/go/src/runtime/panic.go:527 +0x90
runtime.mSpan_Sweep(0x6e2bd40, 0x32900000300, 0xc800071301)
    /usr/local/go/src/runtime/mgcsweep.go:186 +0x800
runtime.sweepone(0x40cf2)
    /usr/local/go/src/runtime/mgcsweep.go:97 +0x154
runtime.gosweepone.func1()
    /usr/local/go/src/runtime/mgcsweep.go:109 +0x21
runtime.systemstack(0xc82001d500)
    /usr/local/go/src/runtime/asm_amd64.s:262 +0x79
runtime.mstart()
    /usr/local/go/src/runtime/proc1.go:674

goroutine 3 [running]:
runtime.systemstack_switch()
    /usr/local/go/src/runtime/asm_amd64.s:216 fp=0xc82002cf58 sp=0xc82002cf50
runtime.gosweepone(0x0)
    /usr/local/go/src/runtime/mgcsweep.go:110 +0x3d fp=0xc82002cf80 sp=0xc82002cf58
runtime.bgsweep(0xc820016070)
    /usr/local/go/src/runtime/mgcsweep.go:54 +0xb6 fp=0xc82002cfb8 sp=0xc82002cf80
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc82002cfc0 sp=0xc82002cfb8
created by runtime.gcenable
    /usr/local/go/src/runtime/mgc.go:206 +0x53

goroutine 1 [IO wait, 215 minutes]:
net.runtime_pollWait(0xfa6f98, 0x72, 0xc820010190)
    /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc8211c8610, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8211c8610, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc8211c85b0, 0x0, 0xfa7090, 0xc92aa360e0)
    /usr/local/go/src/net/fd_unix.go:408 +0x27c
net.(*TCPListener).AcceptTCP(0xc82023c048, 0x5f760, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:254 +0x4d
net/http.tcpKeepAliveListener.Accept(0xc82023c048, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2135 +0x41
net/http.(*Server).Serve(0xc823f8a120, 0xfa7058, 0xc82023c048, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1887 +0xb3
net/http.(*Server).ListenAndServe(0xc823f8a120, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1877 +0x136
net/http.ListenAndServe(0x7fff5fbffc4b, 0x5, 0xfa5fc0, 0xc82000ae70, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1967 +0x8f
main.main()
    /var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/gopath/src/golang.org/x/tools/cmd/godoc/main.go:314 +0x18fd

goroutine 17 [syscall, 696 minutes, locked to thread]:
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1

goroutine 1223 [chan receive, 696 minutes]:
golang.org/x/tools/godoc.(*Corpus).refreshMetadataLoop(0xc8201ae000)
    /var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/gopath/src/golang.org/x/tools/godoc/meta.go:140 +0x3a
created by golang.org/x/tools/godoc.(*Corpus).Init
    /var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/gopath/src/golang.org/x/tools/godoc/corpus.go:145 +0x8a
Tue 10 Nov 2015 03:20:18 CET

rsc · 2015-11-16T18:28:58Z

@thaustad Can you post a copy of the output of $(go list ...) to get a sense of how many packages you have and therefore what the working set is for godoc? I'd like to see if I can reproduce this on my darwin/amd64 machine. Thanks.

thaustad · 2015-11-16T22:21:46Z

My GOPATH is /Users/thomas/code/go:/Users/thomas/code/work/go

archive/tar
archive/zip
bufio
bytes
cmd/addr2line
cmd/api
cmd/asm
cmd/asm/internal/arch
cmd/asm/internal/asm
cmd/asm/internal/flags
cmd/asm/internal/lex
cmd/cgo
cmd/compile
cmd/compile/internal/amd64
cmd/compile/internal/arm
cmd/compile/internal/arm64
cmd/compile/internal/big
cmd/compile/internal/gc
cmd/compile/internal/ppc64
cmd/compile/internal/x86
cmd/cover
cmd/dist
cmd/doc
cmd/fix
cmd/go
cmd/gofmt
cmd/internal/gcprog
cmd/internal/goobj
cmd/internal/obj
cmd/internal/obj/arm
cmd/internal/obj/arm64
cmd/internal/obj/ppc64
cmd/internal/obj/x86
cmd/internal/objfile
cmd/internal/rsc.io/arm/armasm
cmd/internal/rsc.io/x86/x86asm
cmd/link
cmd/link/internal/amd64
cmd/link/internal/arm
cmd/link/internal/arm64
cmd/link/internal/ld
cmd/link/internal/ppc64
cmd/link/internal/x86
cmd/nm
cmd/objdump
cmd/pack
cmd/pprof
cmd/pprof/internal/commands
cmd/pprof/internal/driver
cmd/pprof/internal/fetch
cmd/pprof/internal/plugin
cmd/pprof/internal/profile
cmd/pprof/internal/report
cmd/pprof/internal/svg
cmd/pprof/internal/symbolizer
cmd/pprof/internal/symbolz
cmd/pprof/internal/tempfile
cmd/trace
cmd/vet
cmd/vet/whitelist
cmd/yacc
compress/bzip2
compress/flate
compress/gzip
compress/lzw
compress/zlib
container/heap
container/list
container/ring
crypto
crypto/aes
crypto/cipher
crypto/des
crypto/dsa
crypto/ecdsa
crypto/elliptic
crypto/hmac
crypto/md5
crypto/rand
crypto/rc4
crypto/rsa
crypto/sha1
crypto/sha256
crypto/sha512
crypto/subtle
crypto/tls
crypto/x509
crypto/x509/pkix
database/sql
database/sql/driver
debug/dwarf
debug/elf
debug/gosym
debug/macho
debug/pe
debug/plan9obj
encoding
encoding/ascii85
encoding/asn1
encoding/base32
encoding/base64
encoding/binary
encoding/csv
encoding/gob
encoding/hex
encoding/json
encoding/pem
encoding/xml
errors
expvar
flag
fmt
go/ast
go/build
go/constant
go/doc
go/format
go/importer
go/internal/gccgoimporter
go/internal/gcimporter
go/parser
go/printer
go/scanner
go/token
go/types
hash
hash/adler32
hash/crc32
hash/crc64
hash/fnv
html
html/template
image
image/color
image/color/palette
image/draw
image/gif
image/internal/imageutil
image/jpeg
image/png
index/suffixarray
internal/format
internal/singleflight
internal/testenv
internal/trace
io
io/ioutil
log
log/syslog
math
math/big
math/cmplx
math/rand
mime
mime/multipart
mime/quotedprintable
net
net/http
net/http/cgi
net/http/cookiejar
net/http/fcgi
net/http/httptest
net/http/httputil
net/http/internal
net/http/pprof
net/internal/socktest
net/mail
net/rpc
net/rpc/jsonrpc
net/smtp
net/textproto
net/url
os
os/exec
os/signal
os/user
path
path/filepath
reflect
regexp
regexp/syntax
runtime
runtime/cgo
runtime/debug
runtime/pprof
runtime/race
runtime/trace
sort
strconv
strings
sync
sync/atomic
syscall
testing
testing/iotest
testing/quick
text/scanner
text/tabwriter
text/template
text/template/parse
time
unicode
unicode/utf16
unicode/utf8
unsafe
bitarray
code.google.com/p/appengine-go/appengine
code.google.com/p/appengine-go/appengine/aetest
code.google.com/p/appengine-go/appengine/blobstore
code.google.com/p/appengine-go/appengine/capability
code.google.com/p/appengine-go/appengine/channel
code.google.com/p/appengine-go/appengine/cloudsql
code.google.com/p/appengine-go/appengine/datastore
code.google.com/p/appengine-go/appengine/delay
code.google.com/p/appengine-go/appengine/file
code.google.com/p/appengine-go/appengine/image
code.google.com/p/appengine-go/appengine/log
code.google.com/p/appengine-go/appengine/mail
code.google.com/p/appengine-go/appengine/memcache
code.google.com/p/appengine-go/appengine/module
code.google.com/p/appengine-go/appengine/remote_api
code.google.com/p/appengine-go/appengine/runtime
code.google.com/p/appengine-go/appengine/search
code.google.com/p/appengine-go/appengine/socket
code.google.com/p/appengine-go/appengine/taskqueue
code.google.com/p/appengine-go/appengine/urlfetch
code.google.com/p/appengine-go/appengine/user
code.google.com/p/appengine-go/appengine/xmpp
code.google.com/p/appengine-go/appengine_internal
code.google.com/p/appengine-go/appengine_internal/app_identity
code.google.com/p/appengine-go/appengine_internal/base
code.google.com/p/appengine-go/appengine_internal/blobstore
code.google.com/p/appengine-go/appengine_internal/capability
code.google.com/p/appengine-go/appengine_internal/channel
code.google.com/p/appengine-go/appengine_internal/datastore
code.google.com/p/appengine-go/appengine_internal/files
code.google.com/p/appengine-go/appengine_internal/image
code.google.com/p/appengine-go/appengine_internal/init
code.google.com/p/appengine-go/appengine_internal/log
code.google.com/p/appengine-go/appengine_internal/mail
code.google.com/p/appengine-go/appengine_internal/memcache
code.google.com/p/appengine-go/appengine_internal/modules
code.google.com/p/appengine-go/appengine_internal/remote_api
code.google.com/p/appengine-go/appengine_internal/runtime_config
code.google.com/p/appengine-go/appengine_internal/search
code.google.com/p/appengine-go/appengine_internal/socket
code.google.com/p/appengine-go/appengine_internal/system
code.google.com/p/appengine-go/appengine_internal/taskqueue
code.google.com/p/appengine-go/appengine_internal/urlfetch
code.google.com/p/appengine-go/appengine_internal/user
code.google.com/p/appengine-go/appengine_internal/xmpp
code.google.com/p/appengine-go/example/counter
code.google.com/p/appengine-go/example/delay
code.google.com/p/appengine-go/example/guestbook
code.google.com/p/appengine-go/example/helloworld
code.google.com/p/appengine-go/example/logviewer
code.google.com/p/appengine-go/example/mandelbrot
code.google.com/p/appengine-go/example/module/frontend
code.google.com/p/appengine-go/example/module/id
code.google.com/p/appengine-go/example/module/memcache
code.google.com/p/appengine-go/example/moustachio/moustachio
code.google.com/p/appengine-go/example/moustachio/resize
code.google.com/p/appengine-go/example/remote_api
code.google.com/p/appengine-go/example/transaction
code.google.com/p/appengine-go/go-app-builder
code.google.com/p/appengine-go/goapp
code.google.com/p/appengine-go/gofix
github.com/BurntSushi/xgb
github.com/BurntSushi/xgb/bigreq
github.com/BurntSushi/xgb/composite
github.com/BurntSushi/xgb/damage
github.com/BurntSushi/xgb/dpms
github.com/BurntSushi/xgb/dri2
github.com/BurntSushi/xgb/examples/atoms
github.com/BurntSushi/xgb/examples/create-window
github.com/BurntSushi/xgb/examples/get-active-window
github.com/BurntSushi/xgb/examples/randr
github.com/BurntSushi/xgb/examples/xinerama
github.com/BurntSushi/xgb/ge
github.com/BurntSushi/xgb/glx
github.com/BurntSushi/xgb/randr
github.com/BurntSushi/xgb/record
github.com/BurntSushi/xgb/render
github.com/BurntSushi/xgb/screensaver
github.com/BurntSushi/xgb/shape
github.com/BurntSushi/xgb/shm
github.com/BurntSushi/xgb/xcmisc
github.com/BurntSushi/xgb/xevie
github.com/BurntSushi/xgb/xf86dri
github.com/BurntSushi/xgb/xf86vidmode
github.com/BurntSushi/xgb/xfixes
github.com/BurntSushi/xgb/xgbgen
github.com/BurntSushi/xgb/xinerama
github.com/BurntSushi/xgb/xprint
github.com/BurntSushi/xgb/xproto
github.com/BurntSushi/xgb/xselinux
github.com/BurntSushi/xgb/xtest
github.com/BurntSushi/xgb/xv
github.com/BurntSushi/xgb/xvmc
github.com/bamiaux/rez
github.com/bamiaux/rez/asm
github.com/bamiaux/rez/rezgen
github.com/disintegration/gift
github.com/disintegration/imaging
github.com/fawick/speedtest-resize
github.com/go-av/codec
github.com/go-fsnotify/fsnotify
github.com/go-gl/examples/glfw31-gl21-cube
github.com/go-gl/examples/glfw31-gl41core-cube
github.com/go-gl/gl/all-core/gl
github.com/go-gl/gl/v2.1/gl
github.com/go-gl/gl/v3.2-compatibility/gl
github.com/go-gl/gl/v3.2-core/gl
github.com/go-gl/gl/v3.3-compatibility/gl
github.com/go-gl/gl/v3.3-core/gl
github.com/go-gl/gl/v4.1-compatibility/gl
github.com/go-gl/gl/v4.1-core/gl
github.com/go-gl/gl/v4.4-compatibility/gl
github.com/go-gl/gl/v4.4-core/gl
github.com/go-gl/gl/v4.5-compatibility/gl
github.com/go-gl/gl/v4.5-core/gl
github.com/go-gl/glfw/v3.0/glfw
github.com/go-gl/glfw/v3.1/glfw
github.com/go-gl/glow
github.com/go-gl/mathgl/mgl32
github.com/go-gl/mathgl/mgl32/matstack
github.com/go-gl/mathgl/mgl64
github.com/go-gl/mathgl/mgl64/matstack
github.com/golang/exp/shiny/driver
github.com/golang/exp/shiny/driver/gldriver
github.com/golang/exp/shiny/driver/internal/errscreen
github.com/golang/exp/shiny/driver/internal/pump
github.com/golang/exp/shiny/driver/internal/swizzle
github.com/golang/exp/shiny/driver/windriver
github.com/golang/exp/shiny/driver/x11driver
github.com/golang/exp/shiny/font
github.com/golang/exp/shiny/font/plan9font
github.com/golang/exp/shiny/screen
github.com/golang/exp/shiny/vendor/github.com/BurntSushi/xgb
github.com/golang/exp/shiny/vendor/github.com/BurntSushi/xgb/render
github.com/golang/exp/shiny/vendor/github.com/BurntSushi/xgb/shm
github.com/golang/exp/shiny/vendor/github.com/BurntSushi/xgb/xproto
github.com/golang/lint
github.com/golang/lint/golint
github.com/gopherjs/gopherjs
github.com/gopherjs/gopherjs/build
github.com/gopherjs/gopherjs/compiler
github.com/gopherjs/gopherjs/compiler/analysis
github.com/gopherjs/gopherjs/compiler/astutil
github.com/gopherjs/gopherjs/compiler/filter
github.com/gopherjs/gopherjs/compiler/prelude
github.com/gopherjs/gopherjs/compiler/typesutil
github.com/gopherjs/gopherjs/gcexporter
github.com/gopherjs/gopherjs/js
github.com/gopherjs/gopherjs/nosync
github.com/gopherjs/gopherjs/tests
github.com/gopherjs/webgl
github.com/gordonklaus/ineffassign
github.com/jstemmer/gotags
github.com/kardianos/osext
github.com/kisielk/errcheck
github.com/kisielk/errcheck/internal/errcheck
github.com/kisielk/gotool
github.com/kr/fs
github.com/lib/pq
github.com/lib/pq/hstore
github.com/lib/pq/listen_example
github.com/lib/pq/oid
github.com/nareix/codec
github.com/neelance/sourcemap
github.com/nfnt/resize
github.com/nsf/gocode
github.com/pkg/profile
github.com/rogpeppe/godef
github.com/rogpeppe/godef/go/ast
github.com/rogpeppe/godef/go/parser
github.com/rogpeppe/godef/go/printer
github.com/rogpeppe/godef/go/scanner
github.com/rogpeppe/godef/go/sym
github.com/rogpeppe/godef/go/token
github.com/rogpeppe/godef/go/types
github.com/spf13/cobra
github.com/spf13/pflag
golang.org/x/crypto/bcrypt
golang.org/x/crypto/blowfish
golang.org/x/crypto/bn256
golang.org/x/crypto/cast5
golang.org/x/crypto/curve25519
golang.org/x/crypto/hkdf
golang.org/x/crypto/md4
golang.org/x/crypto/nacl/box
golang.org/x/crypto/nacl/secretbox
golang.org/x/crypto/ocsp
golang.org/x/crypto/openpgp
golang.org/x/crypto/openpgp/armor
golang.org/x/crypto/openpgp/clearsign
golang.org/x/crypto/openpgp/elgamal
golang.org/x/crypto/openpgp/errors
golang.org/x/crypto/openpgp/packet
golang.org/x/crypto/openpgp/s2k
golang.org/x/crypto/otr
golang.org/x/crypto/pbkdf2
golang.org/x/crypto/poly1305
golang.org/x/crypto/ripemd160
golang.org/x/crypto/salsa20
golang.org/x/crypto/salsa20/salsa
golang.org/x/crypto/scrypt
golang.org/x/crypto/sha3
golang.org/x/crypto/ssh
golang.org/x/crypto/ssh/agent
golang.org/x/crypto/ssh/terminal
golang.org/x/crypto/ssh/test
golang.org/x/crypto/twofish
golang.org/x/crypto/xtea
golang.org/x/crypto/xts
golang.org/x/exp/ebnf
golang.org/x/exp/ebnflint
golang.org/x/exp/mmap
golang.org/x/exp/old/netchan
golang.org/x/exp/shiny/driver
golang.org/x/exp/shiny/driver/gldriver
golang.org/x/exp/shiny/driver/internal/errscreen
golang.org/x/exp/shiny/driver/internal/pump
golang.org/x/exp/shiny/driver/internal/swizzle
golang.org/x/exp/shiny/driver/internal/x11key
golang.org/x/exp/shiny/driver/windriver
golang.org/x/exp/shiny/driver/x11driver
golang.org/x/exp/shiny/font
golang.org/x/exp/shiny/font/plan9font
golang.org/x/exp/shiny/screen
golang.org/x/exp/shiny/vendor/github.com/BurntSushi/xgb
golang.org/x/exp/shiny/vendor/github.com/BurntSushi/xgb/render
golang.org/x/exp/shiny/vendor/github.com/BurntSushi/xgb/shm
golang.org/x/exp/shiny/vendor/github.com/BurntSushi/xgb/xproto
golang.org/x/exp/utf8string
golang.org/x/image/bmp
golang.org/x/image/colornames
golang.org/x/image/draw
golang.org/x/image/font
golang.org/x/image/font/basicfont
golang.org/x/image/font/plan9font
golang.org/x/image/math/f32
golang.org/x/image/math/f64
golang.org/x/image/math/fixed
golang.org/x/image/riff
golang.org/x/image/tiff
golang.org/x/image/tiff/lzw
golang.org/x/image/vp8
golang.org/x/image/vp8l
golang.org/x/image/webp
golang.org/x/image/webp/nycbcra
golang.org/x/mobile/app
golang.org/x/mobile/app/internal/apptest
golang.org/x/mobile/app/internal/testapp
golang.org/x/mobile/asset
golang.org/x/mobile/bind
golang.org/x/mobile/bind/java
golang.org/x/mobile/bind/java/testpkg
golang.org/x/mobile/bind/objc
golang.org/x/mobile/bind/objc/testpkg
golang.org/x/mobile/bind/objc/testpkg/go_testpkg
golang.org/x/mobile/bind/seq
golang.org/x/mobile/cmd/gobind
golang.org/x/mobile/cmd/gomobile
golang.org/x/mobile/event/key
golang.org/x/mobile/event/lifecycle
golang.org/x/mobile/event/mouse
golang.org/x/mobile/event/paint
golang.org/x/mobile/event/size
golang.org/x/mobile/event/touch
golang.org/x/mobile/example/audio
golang.org/x/mobile/example/basic
golang.org/x/mobile/example/bind/hello
golang.org/x/mobile/example/network
golang.org/x/mobile/example/sprite
golang.org/x/mobile/exp/app/debug
golang.org/x/mobile/exp/audio
golang.org/x/mobile/exp/audio/al
golang.org/x/mobile/exp/f32
golang.org/x/mobile/exp/font
golang.org/x/mobile/exp/gl/glutil
golang.org/x/mobile/exp/sensor
golang.org/x/mobile/exp/sprite
golang.org/x/mobile/exp/sprite/clock
golang.org/x/mobile/exp/sprite/glsprite
golang.org/x/mobile/exp/sprite/portable
golang.org/x/mobile/geom
golang.org/x/mobile/gl
golang.org/x/mobile/internal/loader
golang.org/x/mobile/internal/mobileinit
golang.org/x/net/context
golang.org/x/net/dict
golang.org/x/net/html
golang.org/x/net/html/atom
golang.org/x/net/html/charset
golang.org/x/net/icmp
golang.org/x/net/idna
golang.org/x/net/internal/iana
golang.org/x/net/internal/nettest
golang.org/x/net/internal/timeseries
golang.org/x/net/ipv4
golang.org/x/net/ipv6
golang.org/x/net/netutil
golang.org/x/net/proxy
golang.org/x/net/publicsuffix
golang.org/x/net/trace
golang.org/x/net/webdav
golang.org/x/net/websocket
golang.org/x/tools/benchmark/parse
golang.org/x/tools/blog
golang.org/x/tools/blog/atom
golang.org/x/tools/cmd/benchcmp
golang.org/x/tools/cmd/callgraph
golang.org/x/tools/cmd/cover
golang.org/x/tools/cmd/digraph
golang.org/x/tools/cmd/eg
golang.org/x/tools/cmd/fiximports
golang.org/x/tools/cmd/godex
golang.org/x/tools/cmd/godoc
golang.org/x/tools/cmd/goimports
golang.org/x/tools/cmd/gomvpkg
golang.org/x/tools/cmd/gorename
golang.org/x/tools/cmd/gotype
golang.org/x/tools/cmd/html2article
golang.org/x/tools/cmd/oracle
golang.org/x/tools/cmd/present
golang.org/x/tools/cmd/ssadump
golang.org/x/tools/cmd/stress
golang.org/x/tools/cmd/stringer
golang.org/x/tools/cmd/tip
golang.org/x/tools/cmd/vet
golang.org/x/tools/cmd/vet/whitelist
golang.org/x/tools/container/intsets
golang.org/x/tools/cover
golang.org/x/tools/go/ast/astutil
golang.org/x/tools/go/buildutil
golang.org/x/tools/go/callgraph
golang.org/x/tools/go/callgraph/cha
golang.org/x/tools/go/callgraph/rta
golang.org/x/tools/go/callgraph/static
golang.org/x/tools/go/exact
golang.org/x/tools/go/gccgoimporter
golang.org/x/tools/go/gcimporter
golang.org/x/tools/go/importer
golang.org/x/tools/go/loader
golang.org/x/tools/go/pointer
golang.org/x/tools/go/ssa
golang.org/x/tools/go/ssa/interp
golang.org/x/tools/go/ssa/ssautil
golang.org/x/tools/go/types
golang.org/x/tools/go/types/typeutil
golang.org/x/tools/go/vcs
golang.org/x/tools/godoc
golang.org/x/tools/godoc/analysis
golang.org/x/tools/godoc/dl
golang.org/x/tools/godoc/proxy
golang.org/x/tools/godoc/redirect
golang.org/x/tools/godoc/short
golang.org/x/tools/godoc/static
golang.org/x/tools/godoc/util
golang.org/x/tools/godoc/vfs
golang.org/x/tools/godoc/vfs/gatefs
golang.org/x/tools/godoc/vfs/httpfs
golang.org/x/tools/godoc/vfs/mapfs
golang.org/x/tools/godoc/vfs/zipfs
golang.org/x/tools/imports
golang.org/x/tools/oracle
golang.org/x/tools/oracle/serial
golang.org/x/tools/playground
golang.org/x/tools/playground/socket
golang.org/x/tools/present
golang.org/x/tools/refactor/eg
golang.org/x/tools/refactor/importgraph
golang.org/x/tools/refactor/rename
golang.org/x/tools/refactor/satisfy
gopherspace
gopherspace/joystixfont
gopkg.in/fsnotify.v1
gopkg.in/yaml.v2
graph
listenall
pgstatus
rsc.io/benchstat
rsc.io/benchstat/internal/go-moremath/internal/mathtest
rsc.io/benchstat/internal/go-moremath/mathx
rsc.io/benchstat/internal/go-moremath/scale
rsc.io/benchstat/internal/go-moremath/stats
rsc.io/benchstat/internal/go-moremath/vec
scgi
templ
xxhash

These are from the second (work) path:

appserver
collector
logoserver
maptile
mczapper
table
tagdata
tcache
xmlwriter

I have removed 55 packages from the first path and 3 from the last. They are all written by me.

Some random notes:

I just noticed that the work path contains a package named "archive" which does not show up, probably because of archive in the stdlib.
github.com/lib/pq exists in both paths (slightly different versions).
I have a similar setup on a linux/amd64 machine which works without issue. There are far fewer packages on that computer. I can send you a diff if you want.

go list ... writes the following to stderr:

can't load package: package github.com/golang/exp/ebnf: code in directory /Users/thomas/code/go/src/github.com/golang/exp/ebnf expects import "golang.org/x/exp/ebnf"
can't load package: package github.com/golang/exp/ebnflint: code in directory /Users/thomas/code/go/src/github.com/golang/exp/ebnflint expects import "golang.org/x/exp/ebnflint"
can't load package: package github.com/golang/exp/old/netchan: code in directory /Users/thomas/code/go/src/github.com/golang/exp/old/netchan expects import "golang.org/x/exp/old/netchan"
can't load package: package github.com/golang/exp/utf8string: code in directory /Users/thomas/code/go/src/github.com/golang/exp/utf8string expects import "golang.org/x/exp/utf8string"

(I had another crash before the weekend that was different. I'll post it shortly.)

thaustad · 2015-11-16T22:29:19Z

Another, but different.

$ulimit -n 999; godoc -analysis="type" -http=:6060; date
2015/11/11 00:08:52 Type analysis complete.
runtime:objectstart Span weird: p=0xc8f7b24460 k=0x647bd92 s.start=0xc8f7b22000 s.limit=0xc8f7b25fe0 s.state=4
fatal error: objectstart: bad pointer in unexpected span
runtime:objectstart Span weird: p=0xc8f7b242d0 k=0x647bd92 s.start=0xc8f7b22000 s.limit=0xc8f7b25fe0 s.state=4
fatal error: objectstart: bad pointer in unexpected span

goroutine 304 [running]:
runtime.throw(0x8a94c0, 0x2b)
    /usr/local/go/src/runtime/panic.go:527 +0x90 fp=0xc82035ce10 sp=0xc82035cdf8
runtime.heapBitsForObject(0xc8f7b24460, 0x0, 0x0, 0xc800000000, 0x4c4da10)
    /usr/local/go/src/runtime/mbitmap.go:217 +0x287 fp=0xc82035ce48 sp=0xc82035ce10
runtime.scanobject(0xc8f7b18d50, 0xc820025220)
    /usr/local/go/src/runtime/mgcmark.go:878 +0x239 fp=0xc82035cf18 sp=0xc82035ce48
runtime.gcDrainUntilPreempt(0xc820025220, 0x7d0)
    /usr/local/go/src/runtime/mgcmark.go:741 +0x152 fp=0xc82035cf50 sp=0xc82035cf18
runtime.gcBgMarkWorker(0xc820024000)
    /usr/local/go/src/runtime/mgc.go:1329 +0x474 fp=0xc82035cfb8 sp=0xc82035cf50
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc82035cfc0 sp=0xc82035cfb8
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1239 +0x93

goroutine 1 [IO wait, 61 minutes]:
net.runtime_pollWait(0x194b430, 0x72, 0xc8200680a0)
    /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc821f82680, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc821f82680, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc821f82620, 0x0, 0x194b528, 0xc93031e060)
    /usr/local/go/src/net/fd_unix.go:408 +0x27c
net.(*TCPListener).AcceptTCP(0xc82025c340, 0x5f760, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:254 +0x4d
net/http.tcpKeepAliveListener.Accept(0xc82025c340, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2135 +0x41
net/http.(*Server).Serve(0xc821f36240, 0x194b4f0, 0xc82025c340, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1887 +0xb3
net/http.(*Server).ListenAndServe(0xc821f36240, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1877 +0x136
net/http.ListenAndServe(0x7fff5fbffc4b, 0x5, 0x194a458, 0xc82006cdb0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1967 +0x8f
main.main()
    /var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/gopath/src/golang.org/x/tools/cmd/godoc/main.go:314 +0x18fd

goroutine 17 [syscall, 1530 minutes, locked to thread]:
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1

goroutine 1348 [chan receive, 1530 minutes]:
golang.org/x/tools/godoc.(*Corpus).refreshMetadataLoop(0xc8201a4000)
    /var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/gopath/src/golang.org/x/tools/godoc/meta.go:140 +0x3a
created by golang.org/x/tools/godoc.(*Corpus).Init
    /var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/gopath/src/golang.org/x/tools/godoc/corpus.go:145 +0x8a

goroutine 203 [running]:
runtime.throw(0x8a94c0, 0x2b)
    /usr/local/go/src/runtime/panic.go:527 +0x90 fp=0xc8202eee18 sp=0xc8202eee00
runtime.heapBitsForObject(0xc8f7b242d0, 0x0, 0x0, 0xc800000000, 0x4c4da10)
    /usr/local/go/src/runtime/mbitmap.go:217 +0x287 fp=0xc8202eee50 sp=0xc8202eee18
runtime.scanobject(0xc8f7b18300, 0xc82001fc20)
    /usr/local/go/src/runtime/mgcmark.go:878 +0x239 fp=0xc8202eef20 sp=0xc8202eee50
runtime.gcDrain(0xc82001fc20, 0x7d0)
    /usr/local/go/src/runtime/mgcmark.go:689 +0xf4 fp=0xc8202eef50 sp=0xc8202eef20
runtime.gcBgMarkWorker(0xc82001ea00)
    /usr/local/go/src/runtime/mgc.go:1320 +0x281 fp=0xc8202eefb8 sp=0xc8202eef50
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8202eefc0 sp=0xc8202eefb8
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1239 +0x93
Thu 12 Nov 2015 01:37:54 CET

rsc · 2015-11-17T05:21:13Z

@thaustad, when you run godoc like this, do you just let it sit there completely idle, or is something actually accessing its web pages?

thaustad · 2015-11-18T00:16:58Z

Completely idle. I do have two Chrome browser tabs pointing at godoc, but those are also idle (at least from what I can tell from the development tools). By the way, Activity Monitor says godoc uses 5.59 GB.

rsc · 2015-11-18T20:48:28Z

I'd like to find this, and I've got a godoc up in the background on my Mac to see what happens, but it's been days and so far nothing. I don't think a fix for this is going to make it into Go 1.5.2. Punting to Go 1.6.

thaustad · 2015-11-18T23:00:09Z

My gut feeling is that if it hasn't happened within 24 hours it won't happen. I restart godoc from time to time to include new packages. I am willing to run a custom godoc if you want to try that.

rsc · 2015-11-23T20:57:37Z

This may be the same as #13372, which seems to be easier to reproduce. We'll try to chase that one for a little while.

bradfitz · 2016-01-14T17:34:29Z

@thaustad, can you reproduce this at tip still, now that #13372 is fixed?

thaustad · 2016-01-19T02:55:24Z

@bradfitz Actually, the bug has not happened since I last reported it. Don't know why. I still use the same Go and OS versions, but there has been some changes to the packages. I can checkout some old source and see if I can reproduce it again.

bradfitz · 2016-01-26T22:32:48Z

I'm going to assume (hope) that this has been fixed somehow in the meantime. Please report if you find otherwise.

ianlancetaylor modified the milestones: Go1.6, Go1.5.2 Oct 8, 2015

rsc changed the title ~~runtime: fatal error: free list corrupted (2)~~ runtime: fatal error: free list corrupted in godoc after hours of idle Nov 17, 2015

msackman mentioned this issue Nov 17, 2015

runtime: fatal error: free list corrupted (3) #13287

Closed

rsc modified the milestones: Go1.6, Go1.5.2 Nov 18, 2015

bradfitz closed this as completed Jan 26, 2016

bdarnell mentioned this issue Mar 27, 2016

stability: crash on malloc "free list corrupted" cockroachdb/cockroach#5592

Closed

golang locked and limited conversation to collaborators Feb 3, 2017

gopherbot added the FrozenDueToAge label Feb 3, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

runtime: fatal error: free list corrupted in godoc after hours of idle #12879

runtime: fatal error: free list corrupted in godoc after hours of idle #12879

thaustad commented Oct 8, 2015

bradfitz commented Oct 8, 2015

ianlancetaylor commented Oct 8, 2015

thaustad commented Oct 8, 2015

aclements commented Oct 9, 2015

thaustad commented Oct 10, 2015

helinwang commented Nov 3, 2015

ianlancetaylor commented Nov 3, 2015

helinwang commented Nov 3, 2015

aclements commented Nov 3, 2015

helinwang commented Nov 3, 2015

ianlancetaylor commented Nov 3, 2015

helinwang commented Nov 3, 2015

ianlancetaylor commented Nov 3, 2015

thaustad commented Nov 10, 2015

rsc commented Nov 16, 2015

thaustad commented Nov 16, 2015

thaustad commented Nov 16, 2015

rsc commented Nov 17, 2015

thaustad commented Nov 18, 2015

rsc commented Nov 18, 2015

thaustad commented Nov 18, 2015

rsc commented Nov 23, 2015

bradfitz commented Jan 14, 2016

thaustad commented Jan 19, 2016

bradfitz commented Jan 26, 2016

runtime: fatal error: free list corrupted in godoc after hours of idle #12879

runtime: fatal error: free list corrupted in godoc after hours of idle #12879

Comments

thaustad commented Oct 8, 2015

bradfitz commented Oct 8, 2015

ianlancetaylor commented Oct 8, 2015

thaustad commented Oct 8, 2015

aclements commented Oct 9, 2015

thaustad commented Oct 10, 2015

helinwang commented Nov 3, 2015

ianlancetaylor commented Nov 3, 2015

helinwang commented Nov 3, 2015

aclements commented Nov 3, 2015

helinwang commented Nov 3, 2015

ianlancetaylor commented Nov 3, 2015

helinwang commented Nov 3, 2015

ianlancetaylor commented Nov 3, 2015

thaustad commented Nov 10, 2015

rsc commented Nov 16, 2015

thaustad commented Nov 16, 2015

thaustad commented Nov 16, 2015

rsc commented Nov 17, 2015

thaustad commented Nov 18, 2015

rsc commented Nov 18, 2015

thaustad commented Nov 18, 2015

rsc commented Nov 23, 2015

bradfitz commented Jan 14, 2016

thaustad commented Jan 19, 2016

bradfitz commented Jan 26, 2016