"broken" since go 1.21

[asottile]

I say "broken" because according to the golang issue this was never intended to work and was merely working by accident: golang/go#65050 (comment)

here's a small reproduction demoing this breakage. I'm running on macos x86_64 which seems to be the most reliable to reproduce for some reason I don't completely understand:

setup:

$ # not shown: running the install steps in the README
$ gopy build -output=hi -vm=python3 github.com/go-python/gopy/_examples/hi
go build -v github.com/go-python/gopy/_examples/hi

--- Processing package: github.com/go-python/gopy/_examples/hi ---

--- building package ---
gopy build -output=hi -vm=python3 github.com/go-python/gopy/_examples/hi
goimports -w hi.go
go build -mod=mod -buildmode=c-shared -o hi_go.so .
/Users/asottile/workspace/z/venv/bin/python3 build.py
CGO_CFLAGS="-I/Library/Frameworks/Python.framework/Versions/3.9/include/python3.9" -fPIC -Ofast
CGO_LDFLAGS="-L/Library/Frameworks/Python.framework/Versions/3.9/lib" "-lpython3.9" -ldl -framework CoreFoundation
go build -mod=mod -buildmode=c-shared -o _hi.cpython-39-darwin.so .
$ gopy build -output=simple -vm=python3 github.com/go-python/gopy/_examples/simple
go build -v github.com/go-python/gopy/_examples/simple

--- Processing package: github.com/go-python/gopy/_examples/simple ---

--- building package ---
gopy build -output=simple -vm=python3 github.com/go-python/gopy/_examples/simple
goimports -w simple.go
go build -mod=mod -buildmode=c-shared -o simple_go.so .
/Users/asottile/workspace/z/venv/bin/python3 build.py
CGO_CFLAGS="-I/Library/Frameworks/Python.framework/Versions/3.9/include/python3.9" -fPIC -Ofast
CGO_LDFLAGS="-L/Library/Frameworks/Python.framework/Versions/3.9/lib" "-lpython3.9" -ldl -framework CoreFoundation
go build -mod=mod -buildmode=c-shared -o _simple.cpython-39-darwin.so .

reproduction

here's a small python script using the compiled output:

(all that really needs to be involved is >1 extension in the same process)

from hi.hi import Hello
from simple.simple import Add

for _ in range(5000):
    Add(2, 2)
    Hello('hi')

boom

(and sometimes the crash is in various other places -- this just seems to be the most common one)

$ python3 t.py 
fatal error: bad sweepgen in refill

goroutine 17 gp=0x1c000006700 m=1 mp=0x1c000048008 [running, locked to thread]:
runtime.throw({0x106e672a0?, 0x0?})
	/Users/asottile/opt/go/src/runtime/panic.go:1067 +0x48 fp=0x1c000056af8 sp=0x1c000056ac8 pc=0x106e12848
runtime.(*mcache).refill(0x10dc925b8, 0x0?)
	/Users/asottile/opt/go/src/runtime/mcache.go:157 +0x20d fp=0x1c000056b38 sp=0x1c000056af8 pc=0x106dbf04d
runtime.(*mcache).nextFree(0x10dc925b8, 0x34)
	/Users/asottile/opt/go/src/runtime/malloc.go:945 +0x85 fp=0x1c000056b70 sp=0x1c000056b38 pc=0x106db9d05
runtime.mallocgc(0x200, 0x106ea7020, 0x1)
	/Users/asottile/opt/go/src/runtime/malloc.go:1161 +0x4cd fp=0x1c000056c10 sp=0x1c000056b70 pc=0x106e0eb8d
runtime.makeslice(0x0?, 0x1c000056c78?, 0x106dbdc85?)
	/Users/asottile/opt/go/src/runtime/slice.go:116 +0x49 fp=0x1c000056c38 sp=0x1c000056c10 pc=0x106e14149
sync.(*Pool).pinSlow(0x106f4bb60)
	/Users/asottile/opt/go/src/sync/pool.go:241 +0x17c fp=0x1c000056cd0 sp=0x1c000056c38 pc=0x106e1ed7c
sync.(*Pool).pin(0x106f4bb60)
	/Users/asottile/opt/go/src/sync/pool.go:220 +0x46 fp=0x1c000056cf0 sp=0x1c000056cd0 pc=0x106e1ebc6
sync.(*Pool).Get(0x106f4bb60)
	/Users/asottile/opt/go/src/sync/pool.go:135 +0x1c fp=0x1c000056d28 sp=0x1c000056cf0 pc=0x106e1e91c
fmt.newPrinter()
	/Users/asottile/opt/go/src/fmt/print.go:152 +0x1e fp=0x1c000056d50 sp=0x1c000056d28 pc=0x106e4475e
fmt.Sprintf({0x106e66215, 0x11}, {0x1c000056dd8, 0x1, 0x1})
	/Users/asottile/opt/go/src/fmt/print.go:238 +0x30 fp=0x1c000056da8 sp=0x1c000056d50 pc=0x106e44b50
github.com/go-python/gopy/_examples/cpkg.Hello({0x1c00009a00a?, 0x10db58ca9?})
	/Users/asottile/workspace/z/go/pkg/mod/github.com/go-python/gopy@v0.4.10/_examples/cpkg/cpkg.go:33 +0x74 fp=0x1c000056e10 sp=0x1c000056da8 pc=0x106e4d874
github.com/go-python/gopy/_examples/hi.Hello(...)
	/Users/asottile/workspace/z/go/pkg/mod/github.com/go-python/gopy@v0.4.10/_examples/hi/hi.go:34
main.hi_Hello(0x7ffbcb6e07e0, 0x0)
	/Users/asottile/workspace/z/hi/hi.go:1708 +0xd8 fp=0x1c000056e60 sp=0x1c000056e10 pc=0x106e553d8
_cgoexp_ffdfbdb0b480_hi_Hello(0x106deb486?)
	_cgo_gotypes.go:2033 +0x1d fp=0x1c000056e80 sp=0x1c000056e60 pc=0x106e59a3d
runtime.cgocallbackg1(0x106e59a20, 0x7ffee9235c08, 0x0)
	/Users/asottile/opt/go/src/runtime/cgocall.go:442 +0x27b fp=0x1c000056f40 sp=0x1c000056e80 pc=0x106db1b7b
runtime.cgocallbackg(0x106e59a20, 0x7ffee9235c08, 0x0)
	/Users/asottile/opt/go/src/runtime/cgocall.go:361 +0x11a fp=0x1c000056f90 sp=0x1c000056f40 pc=0x106db187a
runtime.cgocallbackg(0x106e59a20, 0x7ffee9235c08, 0x0)
	<autogenerated>:1 +0x29 fp=0x1c000056fb8 sp=0x1c000056f90 pc=0x106e1c189
runtime.cgocallback(0x0, 0x0, 0x0)
	/Users/asottile/opt/go/src/runtime/asm_amd64.s:1084 +0xcc fp=0x1c000056fe0 sp=0x1c000056fb8 pc=0x106e19b6c
runtime: g 17: unexpected return pc for runtime.cgocallback called from 0x10db22381
stack: frame={sp:0x1c000056fb8, fp:0x1c000056fe0} stack=[0x1c000056000,0x1c000057000)
0x000001c000056eb8:  0x000001c000056e9f  0x020201c000056f30 
0x000001c000056ec8:  0x0000000106e12e6e <runtime.exitsyscall+0x000000000000014e>  0x000001c000006700 
0x000001c000056ed8:  0x0000000106e59a20 <_cgoexp_ffdfbdb0b480_hi_Hello+0x0000000000000000>  0x0000000000000000 
0x000001c000056ee8:  0x000001c000006700  0x000001c000056ed8 
0x000001c000056ef8:  0x0000000106db1c20 <runtime.cgocallbackg1.deferwrap2+0x0000000000000000>  0x000001c000056ec5 
0x000001c000056f08:  0x00007ffee9237000  0x0000000000000000 
0x000001c000056f18:  0x0000000000000000  0x0000000000000000 
0x000001c000056f28:  0x000001c000056ef8  0x000001c000056f80 
0x000001c000056f38:  0x0000000106db187a <runtime.cgocallbackg+0x000000000000011a>  0x0000000106e59a20 <_cgoexp_ffdfbdb0b480_hi_Hello+0x0000000000000000> 
0x000001c000056f48:  0x00007ffee9235c08  0x0000000000000000 
0x000001c000056f58:  0x000000010db22381  0x000001c000056fe0 
0x000001c000056f68:  0x0000000000000000  0x000001c000006700 
0x000001c000056f78:  0x000001c000048008  0x000001c000056fa8 
0x000001c000056f88:  0x0000000106e1c189 <runtime.cgocallbackg+0x0000000000000029>  0x0000000106e59a20 <_cgoexp_ffdfbdb0b480_hi_Hello+0x0000000000000000> 
0x000001c000056f98:  0x00007ffee9235c08  0x0000000000000000 
0x000001c000056fa8:  0x00007ffee9235b98  0x0000000106e19b6c <runtime.cgocallback+0x00000000000000cc> 
0x000001c000056fb8: <0x0000000106e59a20 <_cgoexp_ffdfbdb0b480_hi_Hello+0x0000000000000000>  0x00007ffee9235c08 
0x000001c000056fc8:  0x0000000000000000  0x0000000000000000 
0x000001c000056fd8:  0x000000010db22381 >0x0000000000000000 
0x000001c000056fe8:  0x0000000000000000  0x0000000000000000 
0x000001c000056ff8:  0x0000000000000000 

goroutine 2 gp=0xc000006c40 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/Users/asottile/opt/go/src/runtime/proc.go:424 +0xce fp=0xc000042fa8 sp=0xc000042f88 pc=0x106e1292e
runtime.goparkunlock(...)
	/Users/asottile/opt/go/src/runtime/proc.go:430
runtime.forcegchelper()
	/Users/asottile/opt/go/src/runtime/proc.go:337 +0xb3 fp=0xc000042fe0 sp=0xc000042fa8 pc=0x106de2473
runtime.goexit({})
	/Users/asottile/opt/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000042fe8 sp=0xc000042fe0 pc=0x106e19da1
created by runtime.init.7 in goroutine 1
	/Users/asottile/opt/go/src/runtime/proc.go:325 +0x1a

goroutine 3 gp=0xc000007180 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/Users/asottile/opt/go/src/runtime/proc.go:424 +0xce fp=0xc000043780 sp=0xc000043760 pc=0x106e1292e
runtime.goparkunlock(...)
	/Users/asottile/opt/go/src/runtime/proc.go:430
runtime.bgsweep(0xc00006c000)
	/Users/asottile/opt/go/src/runtime/mgcsweep.go:277 +0x94 fp=0xc0000437c8 sp=0xc000043780 pc=0x106dce4d4
runtime.gcenable.gowrap1()
	/Users/asottile/opt/go/src/runtime/mgc.go:204 +0x25 fp=0xc0000437e0 sp=0xc0000437c8 pc=0x106dc2e25
runtime.goexit({})
	/Users/asottile/opt/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000437e8 sp=0xc0000437e0 pc=0x106e19da1
created by runtime.gcenable in goroutine 1
	/Users/asottile/opt/go/src/runtime/mgc.go:204 +0x66

goroutine 4 gp=0xc000007340 m=nil [GC scavenge wait]:
runtime.gopark(0xc00006c000?, 0x106e8b7c8?, 0x1?, 0x0?, 0xc000007340?)
	/Users/asottile/opt/go/src/runtime/proc.go:424 +0xce fp=0xc000043f78 sp=0xc000043f58 pc=0x106e1292e
runtime.goparkunlock(...)
	/Users/asottile/opt/go/src/runtime/proc.go:430
runtime.(*scavengerState).park(0x106f51400)
	/Users/asottile/opt/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000043fa8 sp=0xc000043f78 pc=0x106dcbee9
runtime.bgscavenge(0xc00006c000)
	/Users/asottile/opt/go/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc000043fc8 sp=0xc000043fa8 pc=0x106dcc45c
runtime.gcenable.gowrap2()
	/Users/asottile/opt/go/src/runtime/mgc.go:205 +0x25 fp=0xc000043fe0 sp=0xc000043fc8 pc=0x106dc2dc5
runtime.goexit({})
	/Users/asottile/opt/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000043fe8 sp=0xc000043fe0 pc=0x106e19da1
created by runtime.gcenable in goroutine 1
	/Users/asottile/opt/go/src/runtime/mgc.go:205 +0xa5

goroutine 5 gp=0xc000007c00 m=nil [finalizer wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/Users/asottile/opt/go/src/runtime/proc.go:424 +0xce fp=0xc000044620 sp=0xc000044600 pc=0x106e1292e
runtime.runfinq()
	/Users/asottile/opt/go/src/runtime/mfinal.go:193 +0x107 fp=0xc0000447e0 sp=0xc000044620 pc=0x106dc1ea7
runtime.goexit({})
	/Users/asottile/opt/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000447e8 sp=0xc0000447e0 pc=0x106e19da1
created by runtime.createfing in goroutine 1
	/Users/asottile/opt/go/src/runtime/mfinal.go:163 +0x3d
Abort trap: 6

---

aside: I decided to abandon setuptools-golang for this reason 🤷 -- and yeah a clickbaitish video with more details

[ppoage]

Interesting, I could have sworn I had gopy working on go 1.24.1, I recall needing to install go with brew as being on the right arch was important (for apple silicon, that is).

I can't get the example to even build, there is a depreciated build flag.

Setup under a venv:

(Temp) ppoage@Mac Temp % python --version
Python 3.11.11
(Temp) ppoage@Mac Temp % uv pip install pybindgen
Resolved 1 package in 777ms
Prepared 1 package in 131ms
Installed 1 package in 3ms
 + pybindgen==0.22.1
(Temp) ppoage@Mac Temp % uv pip install --upgrade setuptools wheel
Resolved 2 packages in 79ms
Prepared 2 packages in 0.97ms
Installed 2 packages in 19ms
 + setuptools==79.0.1
 + wheel==0.45.1
(Temp) ppoage@Mac Temp % uv pip list
Package    Version
---------- -------
pybindgen  0.22.1
setuptools 79.0.1
wheel      0.45.1
(Temp) ppoage@Mac Temp % go version
go version go1.24.1 darwin/arm64
(Temp) ppoage@Mac Temp % go mod init dummy.com/dum
go: creating new go.mod: module dummy.com/dum
(Temp) ppoage@Mac Temp % go get github.com/go-python/gopy/_examples/hi
go: added github.com/go-python/gopy v0.4.10
(Temp) ppoage@Mac Temp % go install golang.org/x/tools/cmd/goimports@latest
go: downloading golang.org/x/tools v0.32.0
go: downloading golang.org/x/sync v0.13.0
(Temp) ppoage@Mac Temp % go install github.com/go-python/gopy@latest

Build fails with

clang: error: argument '-Ofast' is deprecated; use '-O3 -ffast-math' for the same behavior, or '-O3' to enable only conforming optimizations [-Werror,-Wdeprecated-ofast]

Full log below. Note: I passed the full python venv path as not doing that can break things.

(Temp) ppoage@Mac Temp % gopy build -output=hi -vm=/Applications/Code/Temp/.venv/bin/python3.11 github.com/go-python/gopy/_examples/hi
go build -v github.com/go-python/gopy/_examples/hi

--- Processing package: github.com/go-python/gopy/_examples/hi ---

--- building package ---
gopy build -output=hi -vm=/Applications/Code/Temp/.venv/bin/python3.11 github.com/go-python/gopy/_examples/hi
goimports -w hi.go
go build -mod=mod -buildmode=c-shared -o hi_go.so .
/Applications/Code/Temp/.venv/bin/python3.11 build.py
CGO_CFLAGS="-I/Users/ppoage/.local/share/uv/python/cpython-3.11.11-macos-aarch64-none/include/python3.11" -fPIC -Ofast
CGO_LDFLAGS="-L/Users/ppoage/.local/share/uv/python/cpython-3.11.11-macos-aarch64-none/lib" "-lpython3.11" -ldl -framework CoreFoundation
go build -mod=mod -buildmode=c-shared -o _hi.cpython-311-darwin.so .
cmd had error: exit status 1  output:
# runtime/cgo
clang: error: argument '-Ofast' is deprecated; use '-O3 -ffast-math' for the same behavior, or '-O3' to enable only conforming optimizations [-Werror,-Wdeprecated-ofast]

2025/04/26 12:21:50 error dispatching command: exit status 1