runtime: Go randomly crashes on Windows with no stack trace, just "unknown pc" and memory/registry dump

[jasontconnell]

What version of Go are you using (go version)?

$ go version
go version go1.14.3 windows/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

OS is Windows 10 Pro 10.0.18362

go env Output

$ go env
set GO111MODULE=on
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\jconnell\AppData\Local\go-build
set GOENV=C:\Users\jconnell\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\jconnell\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=c:\go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=c:\go\pkg\tool\windows_amd64
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=C:\Users\jconnell\go\src\github.com\jasontconnell\chashprms\go.mod
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\jconnell\AppData\Local\Temp\go-build801242386=/tmp/go-build -gno-record-gcc-switches
GOROOT/bin/go version: go version go1.14.3 windows/amd64
GOROOT/bin/go tool compile -V: compile version go1.14.3

What did you do?

I don't think the code matters. It has crashed with multiple different programs. But here's an example. It's more in the "running it repeatedly" that breaks it. I've tested it on Go 1.13 and it runs fine. The earliest version that I've tried that showed the error was Go 1.14.3 as displayed in the version section of this post. It is reproducible on Go 1.15 as well.

So my manager was reporting a crash, a memory dump. I wasn't able to reproduce it locally. But then I wrote the simple program

https://play.golang.org/p/KMabbeiz1hW

And a powershell script

For ($i = 0; $i -lt 100; $i++) {
Write-Host $i
.\chashprms "$i is the value"
}

Run go build and run.ps1.

What did you expect to see?

If I run the same code in Docker using the golang:1.15 image, it runs fine. Runs through 100 instances and always prints out the hash. I would expect to see 100 hashes printed. In Go 1.13 it works fine (the latest I tested where it works).

What did you see instead?

In Go 1.14.3 (the earliest that I tested where it crashes), I see a memory and registry dump after a few calls:

Exception 0xc0000005 0x0 0x7ffd4c110fff 0x910000
PC=0x910000

runtime: unknown pc 0x910000
stack: frame={sp:0x84f6a0, fp:0x0} stack=[0x0,0x84ff00)
000000000084f5a0: 000000000098d7c0 000000000084f610
000000000084f5b0: 0000000000000000 000000000084f6a8
000000000084f5c0: 0000000000989710 000000000084f600
000000000084f5d0: 0000000000000000 00007ffd48e60108
000000000084f5e0: 0000000000000000 0000000000963880
000000000084f5f0: 000000000084f738 00007ffd49d194d1
000000000084f600: 00007ffd49d194cb 00007ffd4bf3ba6f
000000000084f610: 00007ffd4bf200d8 0000000000975c50
000000000084f620: 0000000000000005 000000000084f670
000000000084f630: 00007ffd49d17900 00007ffd4bf51810
000000000084f640: 0000000000963880 0000000000000000
000000000084f650: 0000000000050005 00007ffd49d194cb
000000000084f660: 000000000000001b 0000000000000000
000000000084f670: 0000000000000000 00007ffd4bf200d8
000000000084f680: 0000000000000000 00007ffd49ac0000
000000000084f690: 0000000000000001 00007ffd4bf3c734
000000000084f6a0: <0000000000000001 0000000000000000
000000000084f6b0: 00007ffd0000ebb0 000000000084f7a8
000000000084f6c0: 00000000009897e0 000000000084f728
000000000084f6d0: 0000000000250024 00007ffd48e7b816
000000000084f6e0: 000000000098d690 00007ffd4c06c500
000000000084f6f0: 0000000000963880 00007ffd4c06ea5c
000000000084f700: 000000000000094c 00007ffd48deb2d8
000000000084f710: 00007ffd48de9138 00007ffd4c06c528
000000000084f720: 00007ffd4c07ec6a 00007ffd48de0000
000000000084f730: 00007ffd4c070f8c 00007ffd4bfa0d80
000000000084f740: 000000001c1dc789 000000000098d818
000000000084f750: 000000000084fab0 00000000c0000135
000000000084f760: 0000000000000000 0000000000000040
000000000084f770: 0000000000000004 00007ffd4c0852f0
000000000084f780: 0000000000000001 000000000084f900
000000000084f790: 0000000000975c50 00007ffd4bf4e2a8
runtime: unknown pc 0x910000
stack: frame={sp:0x84f6a0, fp:0x0} stack=[0x0,0x84ff00)
000000000084f5a0: 000000000098d7c0 000000000084f610
000000000084f5b0: 0000000000000000 000000000084f6a8
000000000084f5c0: 0000000000989710 000000000084f600
000000000084f5d0: 0000000000000000 00007ffd48e60108
000000000084f5e0: 0000000000000000 0000000000963880
000000000084f5f0: 000000000084f738 00007ffd49d194d1
000000000084f600: 00007ffd49d194cb 00007ffd4bf3ba6f
000000000084f610: 00007ffd4bf200d8 0000000000975c50
000000000084f620: 0000000000000005 000000000084f670
000000000084f630: 00007ffd49d17900 00007ffd4bf51810
000000000084f640: 0000000000963880 0000000000000000
000000000084f650: 0000000000050005 00007ffd49d194cb
000000000084f660: 000000000000001b 0000000000000000
000000000084f670: 0000000000000000 00007ffd4bf200d8
000000000084f680: 0000000000000000 00007ffd49ac0000
000000000084f690: 0000000000000001 00007ffd4bf3c734
000000000084f6a0: <0000000000000001 0000000000000000
000000000084f6b0: 00007ffd0000ebb0 000000000084f7a8
000000000084f6c0: 00000000009897e0 000000000084f728
000000000084f6d0: 0000000000250024 00007ffd48e7b816
000000000084f6e0: 000000000098d690 00007ffd4c06c500
000000000084f6f0: 0000000000963880 00007ffd4c06ea5c
000000000084f700: 000000000000094c 00007ffd48deb2d8
000000000084f710: 00007ffd48de9138 00007ffd4c06c528
000000000084f720: 00007ffd4c07ec6a 00007ffd48de0000
000000000084f730: 00007ffd4c070f8c 00007ffd4bfa0d80
000000000084f740: 000000001c1dc789 000000000098d818
000000000084f750: 000000000084fab0 00000000c0000135
000000000084f760: 0000000000000000 0000000000000040
000000000084f770: 0000000000000004 00007ffd4c0852f0
000000000084f780: 0000000000000001 000000000084f900
000000000084f790: 0000000000975c50 00007ffd4bf4e2a8
rax 0x7ffd48debe7c
rbx 0x7ffd48debe7a
rcx 0x41
rdi 0xffffffffffbadd11
rsi 0x0
rbp 0x84f900
rsp 0x84f6a0
r8 0x0
r9 0x0
r10 0x0
r11 0x94b
r12 0x7ffd4bf20000
r13 0x0
r14 0x7ffd48debe7c
r15 0xc000007a
rip 0x910000
rflags 0x10202
cs 0x33
fs 0x53
gs 0x2b

We are a Windows shop and do .NET dev and these tools I wrote help with that, I would love to be able to keep them in Go :D For now I have to revert them to 1.13ish but I'll be watching this closely. I am available and on the Gopher slack, if you need more information I'll be happy to provide.

[davecheney]

Is there any anti virus software running on that machine that may be modifying or interposing on your actions?

[jasontconnell]

Yes we apparently run "WebRoot SecureAnywhere Endpoint Protection v9.0.28.48". I had wondered that but Go 1.13 was fine so I failed to mention.

[davecheney]

Are you able to confirm if the problem occurs when "WebRoot SecureAnywhere Endpoint Protection v9.0.28.48" is uninstalled or disabled?

[jasontconnell]

I am unable to modify the settings but the real time protection is not enabled. I can try running it on another computer that isn't work controlled.

[dmitshur]

/cc @aclements @randall77 @mknyszek @prattmic per owners.

[jasontconnell]

https://github.com/jasontconnell/chashprms

For reference. I'll be running it on my other computer shortly. Will report back.

[jasontconnell]

Are you able to confirm if the problem occurs when "WebRoot SecureAnywhere Endpoint Protection v9.0.28.48" is uninstalled or disabled?

My home computer is windows 10 pro but a different build. When i ran that with 1.15 it ran fine! So you're on to something @davecheney but I'm updating windows at the moment and will run again when they're the same version.

Edit: this is taking forever. I'll have an update tomorrow.

[jasontconnell]

Home PC Update: It's updated but not the same exact build number. Still runs fine in 1.15.

I have BitDefender on my home PC but that's apparently much better behaved, if an AV is to blame. It may be partially, but I can't say 100% since it works fine in 1.13.

[ianlancetaylor]

Please try running your program with the environment variable GODEBUG=asyncpreemptoff=1.

[alexbrainman]

Please try running your program with the environment variable GODEBUG=asyncpreemptoff=1.

I had problem that was affected by this - #36492 - but I don't see crashes, I see hangs. But, still, you could try, it won't help.

Alex

[jasontconnell]

Please try running your program with the environment variable GODEBUG=asyncpreemptoff=1.

I had problem that was affected by this - #36492 - but I don't see crashes, I see hangs. But, still, you could try, it won't help.

Alex

Very in depth issue! Yeah, unless I'm doing it wrong (possible) @ianlancetaylor the ascyncpreemptoff=1 setting didn't alleviate the crash. I failed to mention that it also hangs and I can't ctrl+c out of it, I have to close the powershell window.

I've now run it in Go 1.13.15 and it runs perfectly. According to webroot the version of the software we're running was released on May 13 of this year, but I don't know when it was installed. This only just started happening this week, and I was running 1.14+ since Feb 26.

I have a question out to IT about when the webroot update might have been pushed, I will edit this comment with pertinent info when I get a response.

Edit: This lines up with when the issues started happening. I can uninstall webroot after talking to IT. I will do that soon.

[jasontconnell]

AV Update: Was able to uninstall Webroot (IT granted me test permissions to remove it temporarily). Of course it runs fine.

Webroot was also causing issues compiling more than once with version 1.13.15. That update 1.1.226 in the screenshot above is most likely to blame, since I don't remember any issues (that weren't pebcak errors) before this week.

We are going to reinstall it and try to allow-list some paths where I and other devs typically use Go programs. That will not be tonight though. I'll be back.

[alexbrainman]

... But, still, you could try, it won't help.

I was meant to say "But, still, you could try, it won't hurt.". Sorry for confusion.

AV Update: Was able to uninstall Webroot (IT granted me test permissions to remove it temporarily). Of course it runs fine.

Looks like you are wining your battles.

Alex

[jasontconnell]

So... I'm not sure if this will require further digging. At least from my end. With the help of IT we re-installed webroot. It has the same Core update from a screenshot earlier that I was pretty sure would have been causing the issue. It seems to behave much better now even with 1.15, with the same program linked earlier, spitting out 100 hashes.

IT's thinking is that I didn't get the latest definitions or something. We don't have real time protection turned on so it just shouldn't have an effect either way. I am clueless as to a possible cause, yet it was consistently "randomly" reproducible.

So I'm currently not able to reproduce the originally reported issue. I've saved this issue so that if it happens again, and this is closed, I'll reference it. I hope it's just done though. I won't close it but someone on the Go team can if deemed appropriate.

Thanks for the help, I effing love Go :)

[jasontconnell]

Hold the phone, it's happening again. Bombarded with other stuff right now. It was fine until I logged back on to our VPN, I'm not sure if that had anything to do with it, probably coincidence.

[bhperry]

I am having very similar issues. Also running WebRoot, but shutting it down doesn't seem to affect anything. Originally thought it was a problem with my cgo setup, but now I've been testing with a super basic program that just prints a character and then exits. Sometimes it works, sometimes it hits the runtime: unknown pc error and hangs.

I've been developing on WSL with an X server to run a game, but I'd really like to just get it compiling natively on windows. I've tried cross-compiling as well from WSL, and get similar results. Sometimes the build works, other times it does not (with zero code changes)

Running go version 1.15.1 amd64 right now. Tried 1.13.15 and 1.14.8, but neither of those worked. 1.13 actually didn't really work at all, couldn't even run go env without it crashing.

[networkimprov]

Another report of this: #41138

[bhperry]

Ok here's something strange. I noticed that an exe I had cross-compiled (and wasn't working) suddenly started working just fine. So I started time stamping builds and saving all of them. They mostly crash when I build them. But when I go back and run them later (an hour or so, but doesn't seem to be consistent), all the sudden they work.

[networkimprov]

try this and post its output (assuming the Windows command prompt)

set GOTRACEBACK=system
your_app

[bhperry]

When I ran in windows cmd it just gave the runtime: unknown pc and stack dump. Ran in an MYSY2 bash terminal and got the following:

GOTRACEBACK=system (MSYS2)

Exception 0xc0000005 0x0 0x7ffb2f6f2fff 0x1beb7f80000
PC=0x1beb7f80000
signal arrived during external code execution

runtime.cgocall(0x6a0040, 0xc000111ee0, 0x3fe0000000000000)
        C:/Go/src/runtime/cgocall.go:133 +0x55 fp=0xc000111eb0 sp=0xc000111e78 pc=0x5347f5
github.com/go-gl/glfw/v3.3/glfw._Cfunc_glfwInit(0x3fd3d3d300000000)
        _cgo_gotypes.go:1435 +0x50 fp=0xc000111ee0 sp=0xc000111eb0 pc=0x642c10
github.com/go-gl/glfw/v3.3/glfw.Init(0x577798, 0x6a6600)
        C:/Users/bhper/go/src/github.com/go-gl/glfw/v3.3/glfw/glfw.go:36 +0x29 fp=0xc000111f20 sp=0xc000111ee0 pc=0x6450a9
github.com/faiface/pixel/pixelgl.Run(0x719028)
        C:/Users/bhper/go/src/github.com/faiface/pixel/pixelgl/run.go:27 +0x3e fp=0xc000111f70 sp=0xc000111f20 pc=0x64ebfe
main.main()
        C:/Users/bhper/go/src/github.com/bhperry/space-miner/main.go:117 +0x34 fp=0xc000111f88 sp=0xc000111f70 pc=0x681294
runtime.main()
        C:/Go/src/runtime/proc.go:204 +0x209 fp=0xc000111fe0 sp=0xc000111f88 pc=0x56ad09
runtime.goexit()
        C:/Go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc000111fe8 sp=0xc000111fe0 pc=0x598b61

goroutine 2 [force gc (idle)]:
runtime.gopark(0x719348, 0x82e9e0, 0x1411, 0x1)
        C:/Go/src/runtime/proc.go:306 +0xfa fp=0xc000045fb0 sp=0xc000045f90 pc=0x56b0da
runtime.goparkunlock(...)
        C:/Go/src/runtime/proc.go:312
runtime.forcegchelper()
        C:/Go/src/runtime/proc.go:255 +0xcd fp=0xc000045fe0 sp=0xc000045fb0 pc=0x56af6d
runtime.goexit()
        C:/Go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc000045fe8 sp=0xc000045fe0 pc=0x598b61
created by runtime.init.6
        C:/Go/src/runtime/proc.go:243 +0x3c

goroutine 3 [GC sweep wait]:
runtime.gopark(0x719348, 0x82eac0, 0x140c, 0x1)
        C:/Go/src/runtime/proc.go:306 +0xfa fp=0xc000047fa8 sp=0xc000047f88 pc=0x56b0da
runtime.goparkunlock(...)
        C:/Go/src/runtime/proc.go:312
runtime.bgsweep(0xc00000c150)
        C:/Go/src/runtime/mgcsweep.go:163 +0xb2 fp=0xc000047fd8 sp=0xc000047fa8 pc=0x5558d2
runtime.goexit()
        C:/Go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc000047fe0 sp=0xc000047fd8 pc=0x598b61
created by runtime.gcenable
        C:/Go/src/runtime/mgc.go:217 +0x67

goroutine 4 [GC scavenge wait]:
runtime.gopark(0x719348, 0x82ebe0, 0x140d, 0x1)
        C:/Go/src/runtime/proc.go:306 +0xfa fp=0xc000055f78 sp=0xc000055f58 pc=0x56b0da
runtime.goparkunlock(...)
        C:/Go/src/runtime/proc.go:312
runtime.bgscavenge(0xc00000c150)
        C:/Go/src/runtime/mgcscavenge.go:265 +0xe5 fp=0xc000055fd8 sp=0xc000055f78 pc=0x5538a5
runtime.goexit()
        C:/Go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc000055fe0 sp=0xc000055fd8 pc=0x598b61
created by runtime.gcenable
        C:/Go/src/runtime/mgc.go:218 +0x89

goroutine 5 [finalizer wait]:
runtime.gopark(0x719348, 0x8618c8, 0x1ffff1410, 0x1)
        C:/Go/src/runtime/proc.go:306 +0xfa fp=0xc000049f58 sp=0xc000049f38 pc=0x56b0da
runtime.goparkunlock(...)
        C:/Go/src/runtime/proc.go:312
runtime.runfinq()
        C:/Go/src/runtime/mfinal.go:175 +0xb2 fp=0xc000049fe0 sp=0xc000049f58 pc=0x54a5f2
runtime.goexit()
        C:/Go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc000049fe8 sp=0xc000049fe0 pc=0x598b61
created by runtime.createfing
        C:/Go/src/runtime/mfinal.go:156 +0x68
rax     0x7ffb2a7b0158
rbx     0x7ffb2a7b0156
rcx     0x77
rdi     0xffffffffffbadd11
rsi     0x0
rbp     0x0
rsp     0xb75e9fed90
r8      0x97c
r9      0x97c
r10     0x97c
r11     0x97c
r12     0xc000007a
r13     0x0
r14     0x7ffb2a7b0158
r15     0x7ffb2f4f0000
rip     0x1beb7f80000
rflags  0x10202
cs      0x33
fs      0x53
gs      0x2b
exit status 2

Running with set GOTRACEBACK=all did work on windows cmd, so here's that as well:

GOTRACEBACK=all (win cmd)

Exception 0xc0000005 0x0 0x7ffb2f6f2fff 0x212ba160000
PC=0x212ba160000
signal arrived during external code execution

github.com/go-gl/glfw/v3.3/glfw._Cfunc_glfwInit(0x3fd3d3d300000000)
        _cgo_gotypes.go:1435 +0x50
github.com/go-gl/glfw/v3.3/glfw.Init(0xaf7798, 0xc26600)
        C:/Users/bhper/go/src/github.com/go-gl/glfw/v3.3/glfw/glfw.go:36 +0x29
github.com/faiface/pixel/pixelgl.Run(0xc99028)
        C:/Users/bhper/go/src/github.com/faiface/pixel/pixelgl/run.go:27 +0x3e
main.main()
        C:/Users/bhper/go/src/github.com/bhperry/space-miner/main.go:117 +0x34
rax     0x7ffb2a7b0158
rbx     0x7ffb2a7b0156
rcx     0x77
rdi     0xffffffffffbadd11
rsi     0x0
rbp     0x0
rsp     0xa1821feb50
r8      0x97c
r9      0x97c
r10     0x97c
r11     0x97c
r12     0xc000007a
r13     0x0
r14     0x7ffb2a7b0158
r15     0x7ffb2f4f0000
rip     0x212ba160000
rflags  0x10202
cs      0x33
fs      0x53
gs      0x2b
exit status 2

[networkimprov]

Ben, that looks like a different problem from the one originally posted here. Can you file a new issue, and include:

1. a complete program that produces the error
2. the logs you posted above

EDIT: also try it with this env var: GODEBUG=asyncpreemptoff=1

[bhperry]

with GODEBUG=asyncpreemptoff=1 I'm just getting the stack dump (same whether or not GOTRACEBACK is set, not sure if that matters)

asyncpreemptoff

Exception 0xc0000005 0x0 0x7ffb2f6f0fff 0x19fc1a40000
PC=0x19fc1a40000

runtime: unknown pc 0x19fc1a40000
stack: frame={sp:0x31b43fe010, fp:0x0} stack=[0x0,0x31b43ff6f0)
00000031b43fdf10:  00000031b43fdf58  00000031b43fdf80
00000031b43fdf20:  00000031b43fdf48  00000031b43fdf40
00000031b43fdf30:  00000031b43fdf44  0000019f9af50000
00000031b43fdf40:  0000000000000000  0000000000000000
00000031b43fdf50:  0000000000000000  0000000000000005
00000031b43fdf60:  00000031b43fe0a8  00007ffb2d2a9f38
00000031b43fdf70:  0000019f9ad84250  00007ffb2f5146ce
00000031b43fdf80:  00007ffb2f4f00e8  0000019f9ad980b0
00000031b43fdf90:  00007ffb2d2a9f32  00000031b43fdfe0
00000031b43fdfa0:  004f0044004e0049  0053005c00530057
00000031b43fdfb0:  0000019f9ad84250  0000000000000000
00000031b43fdfc0:  0000019f9adc3660  006c006400050005
00000031b43fdfd0:  00007ffb2d2a9f32  0000000000000000
00000031b43fdfe0:  00007ffb00000000  00007ffb2f4f00e8
00000031b43fdff0:  0000000000000000  0000000000000000
00000031b43fe000:  0000000000000001  00007ffb2f513783
00000031b43fe010: <0000019f00000001  0000000000000000
00000031b43fe020:  0000000000000000  00000031b43fe118
00000031b43fe030:  0000000000000000  0000000000000000
00000031b43fe040:  0000000000000000  0000000000000000
00000031b43fe050:  0000019f9ad980b0  00007ffb2f640f00
00000031b43fe060:  0000019f9ad84250  00007ffb2f643520
00000031b43fe070:  000000000000097d  00007ffb2c32ccb8
00000031b43fe080:  00007ffb2c32a148  00007ffb2f640f28
00000031b43fe090:  00007ffb2f653d4f  00007ffb2c320000
00000031b43fe0a0:  00007ffb2f645b14  00007ffb2f570aa0
00000031b43fe0b0:  0000000000000000  0000000000000000
00000031b43fe0c0:  0000000000000000  0000000000000000
00000031b43fe0d0:  0000019f9ad97e50  0000000000000040
00000031b43fe0e0:  0000000000000003  00007ffb2f65a3f0
00000031b43fe0f0:  0000000000000001  00000031b43fe300
00000031b43fe100:  0000019f9adc3660  00007ffb2f551448
runtime: unknown pc 0x19fc1a40000
stack: frame={sp:0x31b43fe010, fp:0x0} stack=[0x0,0x31b43ff6f0)
00000031b43fdf10:  00000031b43fdf58  00000031b43fdf80
00000031b43fdf20:  00000031b43fdf48  00000031b43fdf40
00000031b43fdf30:  00000031b43fdf44  0000019f9af50000
00000031b43fdf40:  0000000000000000  0000000000000000
00000031b43fdf50:  0000000000000000  0000000000000005
00000031b43fdf60:  00000031b43fe0a8  00007ffb2d2a9f38
00000031b43fdf70:  0000019f9ad84250  00007ffb2f5146ce
00000031b43fdf80:  00007ffb2f4f00e8  0000019f9ad980b0
00000031b43fdf90:  00007ffb2d2a9f32  00000031b43fdfe0
00000031b43fdfa0:  004f0044004e0049  0053005c00530057
00000031b43fdfb0:  0000019f9ad84250  0000000000000000
00000031b43fdfc0:  0000019f9adc3660  006c006400050005
00000031b43fdfd0:  00007ffb2d2a9f32  0000000000000000
00000031b43fdfe0:  00007ffb00000000  00007ffb2f4f00e8
00000031b43fdff0:  0000000000000000  0000000000000000
00000031b43fe000:  0000000000000001  00007ffb2f513783
00000031b43fe010: <0000019f00000001  0000000000000000
00000031b43fe020:  0000000000000000  00000031b43fe118
00000031b43fe030:  0000000000000000  0000000000000000
00000031b43fe040:  0000000000000000  0000000000000000
00000031b43fe050:  0000019f9ad980b0  00007ffb2f640f00
00000031b43fe060:  0000019f9ad84250  00007ffb2f643520
00000031b43fe070:  000000000000097d  00007ffb2c32ccb8
00000031b43fe080:  00007ffb2c32a148  00007ffb2f640f28
00000031b43fe090:  00007ffb2f653d4f  00007ffb2c320000
00000031b43fe0a0:  00007ffb2f645b14  00007ffb2f570aa0
00000031b43fe0b0:  0000000000000000  0000000000000000
00000031b43fe0c0:  0000000000000000  0000000000000000
00000031b43fe0d0:  0000019f9ad97e50  0000000000000040
00000031b43fe0e0:  0000000000000003  00007ffb2f65a3f0
00000031b43fe0f0:  0000000000000001  00000031b43fe300
00000031b43fe100:  0000019f9adc3660  00007ffb2f551448
rax     0x7ffb2c32d85c
rbx     0x7ffb2c32d85a
rcx     0x41
rdi     0xffffffffffbadd11
rsi     0x0
rbp     0x7ffb2c409f00
rsp     0x31b43fe010
r8      0x0
r9      0x0
r10     0x0
r11     0x97c
r12     0xc000007a
r13     0x0
r14     0x7ffb2c32d85c
r15     0x7ffb2f4f0000
rip     0x19fc1a40000
rflags  0x10206
cs      0x33
fs      0x53
gs      0x2b
exit status 1

Running a super simple program, GOTRACEBACK and GODEBUG don't seem to make a difference. It also fails more inconsistently than a program using cgo, but still fairly regularly

Code

package main

import "fmt"

func main() {
  fmt.Println("s")
}

Crash

$ GODEBUG=asyncpreemptoff=1 GOTRACEBACK=system go run main.go
Exception 0xc0000005 0x0 0x7ffb2f6f0fff 0x166364e0000
PC=0x166364e0000

runtime: unknown pc 0x166364e0000
stack: frame={sp:0x2a6dbfe5e0, fp:0x0} stack=[0x0,0x2a6dbffcc0)
0000002a6dbfe4e0:  0000002a6dbfe528  0000002a6dbfe550
0000002a6dbfe4f0:  0000002a6dbfe518  0000002a6dbfe510
0000002a6dbfe500:  0000002a6dbfe514  0000016611190000
0000002a6dbfe510:  0000000000000000  0000000000000000
0000002a6dbfe520:  0000000000000000  0000000000000005
0000002a6dbfe530:  0000002a6dbfe678  00007ffb2d2a9f38
0000002a6dbfe540:  0000016610fc4210  00007ffb2f5146ce
0000002a6dbfe550:  00007ffb2f4f00e8  0000016610ff9290
0000002a6dbfe560:  00007ffb2d2a9f32  0000002a6dbfe5b0
0000002a6dbfe570:  004f0044004e0049  0053005c00530057
0000002a6dbfe580:  0000016610fc4210  0000000000000000
0000002a6dbfe590:  0000016610fdf990  006c006400050005
0000002a6dbfe5a0:  00007ffb2d2a9f32  0000000000000000
0000002a6dbfe5b0:  00007ffb00000000  00007ffb2f4f00e8
0000002a6dbfe5c0:  0000000000000000  0000000000000000
0000002a6dbfe5d0:  0000000000000001  00007ffb2f513783
0000002a6dbfe5e0: <0000016600000001  0000000000000000
0000002a6dbfe5f0:  0000000000000000  0000002a6dbfe6e8
0000002a6dbfe600:  0000000000000000  0000000000000000
0000002a6dbfe610:  0000000000000000  0000000000000000
0000002a6dbfe620:  0000016610ff9290  00007ffb2f640f00
0000002a6dbfe630:  0000016610fc4210  00007ffb2f643520
0000002a6dbfe640:  000000000000097d  00007ffb2c32ccb8
0000002a6dbfe650:  00007ffb2c32a148  00007ffb2f640f28
0000002a6dbfe660:  00007ffb2f653d4f  00007ffb2c320000
0000002a6dbfe670:  00007ffb2f645b14  00007ffb2f570aa0
0000002a6dbfe680:  0000000000000000  0000000000000000
0000002a6dbfe690:  0000000000000000  0000000000000000
0000002a6dbfe6a0:  0000016610ff4d90  0000000000000040
0000002a6dbfe6b0:  0000000000000003  00007ffb2f65a3f0
0000002a6dbfe6c0:  0000000000000001  0000002a6dbfe900
0000002a6dbfe6d0:  0000016610fdf990  00007ffb2f551448
runtime: unknown pc 0x166364e0000
stack: frame={sp:0x2a6dbfe5e0, fp:0x0} stack=[0x0,0x2a6dbffcc0)
0000002a6dbfe4e0:  0000002a6dbfe528  0000002a6dbfe550
0000002a6dbfe4f0:  0000002a6dbfe518  0000002a6dbfe510
0000002a6dbfe500:  0000002a6dbfe514  0000016611190000
0000002a6dbfe510:  0000000000000000  0000000000000000
0000002a6dbfe520:  0000000000000000  0000000000000005
0000002a6dbfe530:  0000002a6dbfe678  00007ffb2d2a9f38
0000002a6dbfe540:  0000016610fc4210  00007ffb2f5146ce
0000002a6dbfe550:  00007ffb2f4f00e8  0000016610ff9290
0000002a6dbfe560:  00007ffb2d2a9f32  0000002a6dbfe5b0
0000002a6dbfe570:  004f0044004e0049  0053005c00530057
0000002a6dbfe580:  0000016610fc4210  0000000000000000
0000002a6dbfe590:  0000016610fdf990  006c006400050005
0000002a6dbfe5a0:  00007ffb2d2a9f32  0000000000000000
0000002a6dbfe5b0:  00007ffb00000000  00007ffb2f4f00e8
0000002a6dbfe5c0:  0000000000000000  0000000000000000
0000002a6dbfe5d0:  0000000000000001  00007ffb2f513783
0000002a6dbfe5e0: <0000016600000001  0000000000000000
0000002a6dbfe5f0:  0000000000000000  0000002a6dbfe6e8
0000002a6dbfe600:  0000000000000000  0000000000000000
0000002a6dbfe610:  0000000000000000  0000000000000000
0000002a6dbfe620:  0000016610ff9290  00007ffb2f640f00
0000002a6dbfe630:  0000016610fc4210  00007ffb2f643520
0000002a6dbfe640:  000000000000097d  00007ffb2c32ccb8
0000002a6dbfe650:  00007ffb2c32a148  00007ffb2f640f28
0000002a6dbfe660:  00007ffb2f653d4f  00007ffb2c320000
0000002a6dbfe670:  00007ffb2f645b14  00007ffb2f570aa0
0000002a6dbfe680:  0000000000000000  0000000000000000
0000002a6dbfe690:  0000000000000000  0000000000000000
0000002a6dbfe6a0:  0000016610ff4d90  0000000000000040
0000002a6dbfe6b0:  0000000000000003  00007ffb2f65a3f0
0000002a6dbfe6c0:  0000000000000001  0000002a6dbfe900
0000002a6dbfe6d0:  0000016610fdf990  00007ffb2f551448
rax     0x7ffb2c32d85c
rbx     0x7ffb2c32d85a
rcx     0x41
rdi     0xffffffffffbadd11
rsi     0x0
rbp     0x7ffb2c409f00
rsp     0x2a6dbfe5e0
r8      0x0
r9      0x0
r10     0x0
r11     0x97c
r12     0xc000007a
r13     0x0
r14     0x7ffb2c32d85c
r15     0x7ffb2f4f0000
rip     0x166364e0000
rflags  0x10206
cs      0x33
fs      0x53
gs      0x2b

Is that more related to this issue? Happy to open a new one if you think I should.

[networkimprov]

Hopefully we'll get some pointers from the runtime team on investigating these reports...

cc @aclements @randall77 @prattmic

[networkimprov]

We have confirmation in #41138 that Webroot causes problems for Go. Uninstalling Webroot is a solution.

EDIT: If that fixes your problems, could you retitle the issue to identify Webroot?

[bhperry]

Well that's annoying. I tried turning off webroot protection and it still had the problem, but I didn't notice that webroot still has some background processes up even when it is "off".

I've suspected webroot of doing other crummy things in the past too (like slowing down my old laptop to the point where it was basically useless after the license expired). Guess it's time to find a different AV.

I'll try uninstalling and see what happens.

[bhperry]

wow... uninstalled Webroot and now everything works perfectly. Thanks for the help!

[davecheney]

Thanks for confirming.

[jasontconnell]

Apologies! Yeah our IT team allowed me an exception on webroot. Now we're in a transition to sophos. Hopefully that one behaves better. I've had 1.13 installed in a c:\go-lock folder and have been building my programs that get distributed to other team members since then with that version. I haven't heard any issues since. So, confirming a few months later that uninstalling webroot fixed my issue.

…

On Wed, Sep 9, 2020, 9:19 PM Liam ***@***.***> wrote: Dave, the issue was opened by @jasontconnell <https://github.com/jasontconnell> who has yet to confirm that his problem is fixed. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#40878 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABUVYYHFZW65QNCWFUJ243SFASTXANCNFSM4QEM3H3Q> .

[jimitheat]

Holy fucking shit... I have web root as well and I was going crazy. I upgraded from 1.15 to 1.16 and it failed after installing the whole OS.

I've just turned off webroot and it compiles. OMG I was going nuts.

[bhperry]

I know the feels

[jasontconnell]

Months later follow up part 2. We've switched over to Sophos for our AV. After much revolt by the whole dev team due to performance issues, and it deleting apps I've written in Go, I am able to confirm no issues with Go 1.16 and Sophos. Glad you found this @jimitheat and for @davecheney thinking it could be AV.

Man I hate AV, at the beginning of our Sophos conversion last week, it was scanning every executable before it allowed it to execute, so saving a file in VS Code took like 5-6 seconds since it's doing gofmt and the linting and all that. Brutal.

[JacomusP]

@jasontconnell Hey do you have any other AV suggestions other than Sophos? Or does anybody else have some suggestions?

[alexbrainman]

Hey do you have any other AV suggestions other than Sophos? Or does anybody else have some suggestions?

I use Microsoft Defender. Every copy of Windows 10 come with this antivirus preinstalled. IMHO Microsoft Defender is the least painful antivirus to use.

Still I disable the antivirus when developing. Makes my computer run quicker.

Alex