arm64: fatal error: found bad pointer in Go heap

[kattakarthik]

What version of Go are you using (go version)?

$ go version
go version go1.12.7 linux/arm64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GOARCH="arm64"
GOBIN=""
GOCACHE="/home/ubuntu/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/ubuntu/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_arm64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build994090878=/tmp/go-build -gno-record-gcc-switches"


minio_log.txt
Stack trace

runtime: pointer 0x401a000000 to unallocated span span.base()=0x4009fae000 span.limit=0x4009fb0000 span.state=0

runtime: found in object at *(0x401a6a3030+0xb0)

object=0x401a6a3030 s.base()=0x401a69e000 s.limit=0x401a6a6000 s.spanclass=0 s.elemsize=8192 s.state=mSpanManual

*(object+0) = 0x12

*(object+8) = 0x4002156300

*(object+16) = 0x4

*(object+24) = 0x2

*(object+32) = 0x0

*(object+40) = 0x4009146840

*(object+48) = 0x0

*(object+56) = 0x40091467e0

*(object+64) = 0x0

*(object+72) = 0x0

*(object+80) = 0x0

*(object+88) = 0x401a6a30d8

*(object+96) = 0x75058

*(object+104) = 0x3

*(object+112) = 0x3

*(object+120) = 0x4002742318

*(object+128) = 0x12

*(object+136) = 0x401a6a30f8

*(object+144) = 0x0

*(object+152) = 0x0

*(object+160) = 0x40010eba07

*(object+168) = 0x401a6a3118

*(object+176) = 0x401a58e900 <==

*(object+184) = 0x0

*(object+192) = 0x0

*(object+200) = 0x4008ada9c0

*(object+208) = 0x4002156300

*(object+216) = 0x0

*(object+224) = 0x4009079900

*(object+232) = 0x4009146840

*(object+240) = 0x0

*(object+248) = 0x0

*(object+256) = 0x40091467e0

*(object+264) = 0x4002156300

*(object+272) = 0x401a6a30c0

*(object+280) = 0x1

*(object+288) = 0x0

*(object+296) = 0x0

*(object+304) = 0x40269f26c0

*(object+312) = 0x0

*(object+320) = 0x1

*(object+328) = 0x0

*(object+336) = 0x0

*(object+344) = 0x0

*(object+352) = 0x0

*(object+360) = 0x0

*(object+368) = 0x0

*(object+376) = 0x0

*(object+384) = 0x4009146840

*(object+392) = 0x401a6a30e0

*(object+400) = 0x1

*(object+408) = 0x0

*(object+416) = 0x0

*(object+424) = 0x0

*(object+432) = 0x0

*(object+440) = 0x0

*(object+448) = 0x0

*(object+456) = 0x0

*(object+464) = 0x0

*(object+472) = 0x0

*(object+480) = 0x0

*(object+488) = 0x0

*(object+496) = 0x0

*(object+504) = 0x401a6a34a8

*(object+512) = 0x2a32e0

*(object+520) = 0x4002742240

*(object+528) = 0x4008ada9c0

*(object+536) = 0x0

*(object+544) = 0x0

*(object+552) = 0x0

*(object+560) = 0x40084850a0

*(object+568) = 0x12

*(object+576) = 0x0

*(object+584) = 0x4002742240

*(object+592) = 0x0

*(object+600) = 0x0

*(object+608) = 0x1

*(object+616) = 0x1

*(object+624) = 0x14

*(object+632) = 0x0

*(object+640) = 0x56

*(object+648) = 0x401a6a32f8

*(object+656) = 0x4

*(object+664) = 0xc

*(object+672) = 0x58

*(object+680) = 0x401a6a3308

*(object+688) = 0x164fe40

*(object+696) = 0x0

*(object+704) = 0x401a6a7880

*(object+712) = 0x4008485080

*(object+720) = 0x4008ada9c0

*(object+728) = 0x0

*(object+736) = 0x60dd74

*(object+744) = 0x4008dc02a4

*(object+752) = 0x4009146780

*(object+760) = 0x40010eba00

*(object+768) = 0x4008dc0240

*(object+776) = 0x4009079900

*(object+784) = 0x4002742240

*(object+792) = 0x134f75f

*(object+800) = 0x1b7e8

*(object+808) = 0x401a6a3418

*(object+816) = 0x40002c6058

*(object+824) = 0x56

*(object+832) = 0x401a6a7880

*(object+840) = 0xba

*(object+848) = 0x10b0280

*(object+856) = 0x4000bf4180

*(object+864) = 0x2d5fe0

*(object+872) = 0x40012319e0

*(object+880) = 0x1b7e8

*(object+888) = 0x4009146780

*(object+896) = 0x4008dc0240

*(object+904) = 0x401a6a3468

*(object+912) = 0x4008dc0240

*(object+920) = 0x401a6a33c8

*(object+928) = 0x4008dc0240

*(object+936) = 0xba

*(object+944) = 0x0

*(object+952) = 0x40010eba00

*(object+960) = 0x4

*(object+968) = 0x40084850a0

*(object+976) = 0x12

*(object+984) = 0x0

*(object+992) = 0x1bda8

*(object+1000) = 0x401a6a34d8

*(object+1008) = 0x229b0

*(object+1016) = 0x4008f91760

...

fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)
runtime stack:

runtime.throw(0x13a6c9e, 0x3e)

/usr/local/go/src/runtime/panic.go:617 +0x54 fp=0x4003cbb740 sp=0x4003cbb710 pc=0x3d774

runtime.findObject(0x401a000000, 0x401a6a3030, 0xb0, 0xffff88ac7e20, 0x400005a670, 0x16)

/usr/local/go/src/runtime/mbitmap.go:397 +0x370 fp=0x4003cbb790 sp=0x4003cbb740 pc=0x23f40

runtime.scanblock(0x401a6a3030, 0x1f8, 0x1563a68, 0x400005a670, 0x4003cbbca8)

/usr/local/go/src/runtime/mgcmark.go:1070 +0xa0 fp=0x4003cbb7f0 sp=0x4003cbb790 pc=0x2f740

runtime.scanframeworker(0x4003cbbb90, 0x4003cbbca8, 0x400005a670)

/usr/local/go/src/runtime/mgcmark.go:806 +0x16c fp=0x4003cbb880 sp=0x4003cbb7f0 pc=0x2efac

runtime.scanstack.func1(0x4003cbbb90, 0x0, 0x241c400)

/usr/local/go/src/runtime/mgcmark.go:708 +0x38 fp=0x4003cbb8b0 sp=0x4003cbb880 pc=0x67cd8

runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0x400124fb00, 0x0, 0x0, 0x7fffffff, 0x4003cbbc90, 0x0, 0x0, ...)

/usr/local/go/src/runtime/traceback.go:341 +0xf28 fp=0x4003cbbbf0 sp=0x4003cbb8b0 pc=0x61618

runtime.scanstack(0x400124fb00, 0x400005a670)

/usr/local/go/src/runtime/mgcmark.go:711 +0xf4 fp=0x4003cbbdf0 sp=0x4003cbbbf0 pc=0x2e9b4

runtime.scang(0x400124fb00, 0x400005a670)

/usr/local/go/src/runtime/proc.go:888 +0x1cc fp=0x4003cbbe50 sp=0x4003cbbdf0 pc=0x4137c

runtime.markroot.func1()

/usr/local/go/src/runtime/mgcmark.go:221 +0x64 fp=0x4003cbbe90 sp=0x4003cbbe50 pc=0x67bc4

runtime.markroot(0x400005a670, 0x4000000b5b)

/usr/local/go/src/runtime/mgcmark.go:202 +0x308 fp=0x4003cbbf20 sp=0x4003cbbe90 pc=0x2d968

runtime.gcDrain(0x400005a670, 0x3)

/usr/local/go/src/runtime/mgcmark.go:899 +0x110 fp=0x4003cbbf80 sp=0x4003cbbf20 pc=0x2f0e0

runtime.gcBgMarkWorker.func2()

/usr/local/go/src/runtime/mgc.go:1903 +0x70 fp=0x4003cbbfc0 sp=0x4003cbbf80 pc=0x679f0

runtime.systemstack(0x0)

/usr/local/go/src/runtime/asm_arm64.s:237 +0xa0 fp=0x4003cbbfd0 sp=0x4003cbbfc0 pc=0x6a6d0

runtime.mstart()

/usr/local/go/src/runtime/proc.go:1153 fp=0x4003cbbfd0 sp=0x4003cbbfd0 pc=0x41e10

[kattakarthik]

Seems like same issue as #21568.. But reproducible on latest available release.. Please suggest. This is a blocker for us.

[erifan]

Hi, can you tell us what did you do ? Or is there any way we can reproduce this error ?

[kattakarthik]

@erifan we were testing file uploads using minio. Started minio server(which was built using the above go version) on arm64.
This is reproducible easily (twice in a day or so) for us..

[ianlancetaylor]

This error indicates some sort of memory corruption. By far the most common causes are incorrect uses of unsafe.Pointer, incorrect uses of cgo when calling C code, and incorrect C code.

Can you give us complete and precise instructions for how we can reproduce the problem? Thanks.

[FU100]

Encounter the same problem while uploading file,
or crash with 'fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)'
or crash with 'fatal error: sweep increased allocation count'

crash every few days.

go version go1.12.9 darwin/amd64

build.sh
CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -v -o main main.go

server info
~ # uname -a
Linux (none) 4.9.37 #1 SMP Mon Jun 10 11:28:59 CST 2019 aarch64 GNU/Linux

My go program is a http server, it receives file posted by a http client and then send it to a file server(seaweedfs), this program crash every few days.

fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

runtime stack:
runtime.throw(0x97084b, 0x3e)
/usr/local/go/src/runtime/panic.go:617 +0x54 fp=0x40005dd740 sp=0x40005dd710 pc=0x3d154
runtime.findObject(0x4002dd0000, 0x40014cef18, 0x50, 0x40005dd7f8, 0x40005dd7f0, 0x9ada1c)
/usr/local/go/src/runtime/mbitmap.go:397 +0x370 fp=0x40005dd790 sp=0x40005dd740 pc=0x23c80
runtime.scanblock(0x40014cef18, 0xb0, 0x9d635b, 0x4000041c70, 0x40005ddca8)
/usr/local/go/src/runtime/mgcmark.go:1070 +0xa0 fp=0x40005dd7f0 sp=0x40005dd790 pc=0x2f480
runtime.scanframeworker(0x40005ddb90, 0x40005ddca8, 0x4000041c70)
/usr/local/go/src/runtime/mgcmark.go:806 +0x16c fp=0x40005dd880 sp=0x40005dd7f0 pc=0x2ecec
runtime.scanstack.func1(0x40005ddb90, 0x0, 0x10e9000)
/usr/local/go/src/runtime/mgcmark.go:708 +0x38 fp=0x40005dd8b0 sp=0x40005dd880 pc=0x67828
runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0x4000631e00, 0x0, 0x0, 0x7fffffff, 0x40005ddc90, 0x0, 0x0, ...)
/usr/local/go/src/runtime/traceback.go:341 +0xf28 fp=0x40005ddbf0 sp=0x40005dd8b0 pc=0x61208
runtime.scanstack(0x4000631e00, 0x4000041c70)
/usr/local/go/src/runtime/mgcmark.go:711 +0xf4 fp=0x40005dddf0 sp=0x40005ddbf0 pc=0x2e6f4
runtime.scang(0x4000631e00, 0x4000041c70)
/usr/local/go/src/runtime/proc.go:888 +0x1cc fp=0x40005dde50 sp=0x40005dddf0 pc=0x40d5c
runtime.markroot.func1()
/usr/local/go/src/runtime/mgcmark.go:221 +0x64 fp=0x40005dde90 sp=0x40005dde50 pc=0x67714
runtime.markroot(0x4000041c70, 0x40000000ad)
/usr/local/go/src/runtime/mgcmark.go:202 +0x308 fp=0x40005ddf20 sp=0x40005dde90 pc=0x2d6a8
runtime.gcDrain(0x4000041c70, 0x7)
/usr/local/go/src/runtime/mgcmark.go:899 +0x110 fp=0x40005ddf80 sp=0x40005ddf20 pc=0x2ee20
runtime.gcBgMarkWorker.func2()
/usr/local/go/src/runtime/mgc.go:1926 +0x18c fp=0x40005ddfc0 sp=0x40005ddf80 pc=0x6765c
runtime.systemstack(0x400003be00)
/usr/local/go/src/runtime/asm_arm64.s:237 +0xa0 fp=0x40005ddfd0 sp=0x40005ddfc0 pc=0x6a110
runtime.mstart()
/usr/local/go/src/runtime/proc.go:1153 fp=0x40005ddfd0 sp=0x40005ddfd0 pc=0x417f0

runtime: nelems=170 nalloc=61 previous allocCount=60 nfreed=65535
fatal error: I0906 22:11:03.276269 23937 fileService.go:425] uploadFile: end id = 1169976306616541185, tm = 12ms, size = 21kb, count = 1
I0906 22:11:03.276640 23937 fileService.go:425] uploadFile: end id = 1169976306645901312, tm = 6ms, size = 6kb, count = 1
sweep increased allocation count
runtime: nelems=128 nalloc=18 previous allocCount=17 nfreed=65535
fatal error: sweep increased allocation count
runtime: nelems=512 nalloc=392 previous allocCount=391 nfreed=65535
fatal error: sweep increased allocation count

goroutine 3 [running]:
runtime.throw(0x95ed1a, 0x20)
/usr/local/go/src/runtime/panic.go:617 +0x54 fp=0x400004f610 sp=0x400004f5e0 pc=0x3d154
runtime.(*mspan).sweep(0x7f92a981d8, 0x0, 0x400004f700)
/usr/local/go/src/runtime/mgcsweep.go:329 +0x894 fp=0x400004f730 sp=0x400004f610 pc=0x31d24
runtime.sweepone(0x97f308)
/usr/local/go/src/runtime/mgcsweep.go:136 +0x308 fp=0x400004f7a0 sp=0x400004f730 pc=0x31198
runtime.bgsweep(0x400006e000)
/usr/local/go/src/runtime/mgcsweep.go:73 +0xe4 fp=0x400004f7d0 sp=0x400004f7a0 pc=0x30da4
runtime.goexit()
/usr/local/go/src/runtime/asm_arm64.s:1128 +0x4 fp=0x400004f7d0 sp=0x400004f7d0 pc=0x6c7e4
created by runtime.gcenable
/usr/local/go/src/runtime/mgc.go:208 +0x54

[harshavardhana]

Thanks for reproducing this issue @FU100 - would you mind sharing main.go?

[ianlancetaylor]

@FU100 Unless you are running the same program, this is extremely unlikely to have the same cause as the original report on this issue. Please open a new issue. Thanks.

As noted above, the most likely causes of this kind of error are incorrect uses of unsafe.Pointer, incorrect uses of cgo when calling C code, and incorrect C code.

In the new issue, please give us complete and precise instructions for how we can reproduce the problem. Thanks.