cmd/compile: internal compiler error instantiating generic type

[dsnet]

I apologize for all the bug reports, but I'm hitting another internal compiler error.

$ go.tip version
go version devel go1.18-d9244f8b64 Fri Sep 3 21:31:30 2021 +0000 linux/amd64

$ go.tip test
# github.com/go-json-experiment/json_test [github.com/go-json-experiment/json.test]
<autogenerated>:1: internal compiler error: Dictionary should have already been generated: github.com/go-json-experiment/json..dict.(*arshalers[json.UnmarshalOptions,json.Decoder]).lookup

goroutine 1 [running]:
runtime/debug.Stack()
	/usr/local/go.tip/src/runtime/debug/stack.go:24 +0x65
cmd/compile/internal/base.FatalfAt({0x9df580, 0xc0}, {0xd120ee, 0x34}, {0xc000589320, 0x2, 0x2})
	/usr/local/go.tip/src/cmd/compile/internal/base/print.go:227 +0x154
cmd/compile/internal/base.Fatalf(...)
	/usr/local/go.tip/src/cmd/compile/internal/base/print.go:196
cmd/compile/internal/reflectdata.getDictionary(0xc000732eb0, {0xc0006b1c00, 0x2, 0x2})
	/usr/local/go.tip/src/cmd/compile/internal/reflectdata/reflect.go:2043 +0x245
cmd/compile/internal/reflectdata.methodWrapper(0xc000730070, 0xc000732f00, 0xf5)
	/usr/local/go.tip/src/cmd/compile/internal/reflectdata/reflect.go:1909 +0x10ea
cmd/compile/internal/reflectdata.methods(0xc000730070)
	/usr/local/go.tip/src/cmd/compile/internal/reflectdata/reflect.go:347 +0x39f
cmd/compile/internal/reflectdata.uncommonSize(...)
	/usr/local/go.tip/src/cmd/compile/internal/reflectdata/reflect.go:79
cmd/compile/internal/reflectdata.dcommontype(0xc0006dbf80, 0xc000730070)
	/usr/local/go.tip/src/cmd/compile/internal/reflectdata/reflect.go:716 +0x1cc
cmd/compile/internal/reflectdata.writeType(0xc0006b29a0)
	/usr/local/go.tip/src/cmd/compile/internal/reflectdata/reflect.go:1140 +0x327
cmd/compile/internal/reflectdata.WriteRuntimeTypes()
	/usr/local/go.tip/src/cmd/compile/internal/reflectdata/reflect.go:1254 +0x3d8
cmd/compile/internal/gc.dumpdata()
	/usr/local/go.tip/src/cmd/compile/internal/gc/obj.go:118 +0x6e
cmd/compile/internal/gc.Main(0xd1dcb0)
	/usr/local/go.tip/src/cmd/compile/internal/gc/main.go:327 +0x10de
main.main()
	/usr/local/go.tip/src/cmd/compile/main.go:55 +0xdd

FAIL	github.com/go-json-experiment/json [build failed]

Reproduction:

$ git clone https://github.com/go-json-experiment/json.git
$ cd json
$ git checkout origin/generics-bug4
$ go test

Based on @mdempsky's comment in #48016 (comment), I tried building this with GOEXPERIMENT=unified and that also panics, but due to a different problem:

$ GOEXPERIMENT=unified go.tip test
# github.com/go-json-experiment/json_test [github.com/go-json-experiment/json.test]
fatal error: all goroutines are asleep - deadlock!

goroutine 1 [semacquire]:
sync.runtime_SemacquireMutex(0xc0000cf068, 0x98, 0xa2)
	/usr/local/go.tip/src/runtime/sema.go:71 +0x25
sync.(*Mutex).lockSlow(0xc0004ade4c)
	/usr/local/go.tip/src/sync/mutex.go:138 +0x165
sync.(*Mutex).Lock(...)
	/usr/local/go.tip/src/sync/mutex.go:81
sync.(*Once).doSlow(0xb70e27, 0xc0000cf0e0)
	/usr/local/go.tip/src/sync/once.go:64 +0x53
sync.(*Once).Do(...)
	/usr/local/go.tip/src/sync/once.go:59
cmd/compile/internal/types2.(*Named).load(0xc0004addd0)
	/usr/local/go.tip/src/cmd/compile/internal/types2/named.go:56 +0xdb
cmd/compile/internal/types2.(*Named).TParams(...)
	/usr/local/go.tip/src/cmd/compile/internal/types2/named.go:124
cmd/compile/internal/types2.(*typeWriter).typ(0xc0000cf398, {0xe50750, 0xc0004addd0})
	/usr/local/go.tip/src/cmd/compile/internal/types2/typestring.go:237 +0x3bb
cmd/compile/internal/types2.(*typeWriter).typeList(0xc0000cf398, {0xc0003daec0, 0x2, 0x138b1e0})
	/usr/local/go.tip/src/cmd/compile/internal/types2/typestring.go:272 +0xd2
cmd/compile/internal/types2.typeHash({0xe50750, 0xc0004ade60}, {0xc0003daec0, 0x10, 0x2})
	/usr/local/go.tip/src/cmd/compile/internal/types2/subst.go:269 +0xef
cmd/compile/internal/types2.(*Checker).instance(0xc0000f4640, {0x0, 0x0, 0x0}, {0xe50750, 0xc0004ade60}, {0xc0003daec0, 0xc0005a9e90, 0xc0005d7bc0})
	/usr/local/go.tip/src/cmd/compile/internal/types2/instantiate.go:128 +0x31f
cmd/compile/internal/types2.Instantiate(0xc0005d7bc0, {0xe50750, 0xc0004ade60}, {0xc0003daec0, 0x2, 0x2}, 0x0)
	/usr/local/go.tip/src/cmd/compile/internal/types2/instantiate.go:53 +0xb1
cmd/compile/internal/noder.(*reader2).doTyp(0xc0005d7bc0)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:231 +0x150
cmd/compile/internal/noder.(*pkgReader2).typIdx(0xc0000e0900, {0xc0005d7b60, 0xb}, 0xc0005cc4e0)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:206 +0x175
cmd/compile/internal/noder.(*reader2).typ(0xc0005d7b60)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:178 +0x38
cmd/compile/internal/noder.(*reader2).doTyp(0xc0005d7b60)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:248 +0x7d
cmd/compile/internal/noder.(*pkgReader2).typIdx(0xc0000e0900, {0xc0005d7680, 0x9}, 0xc0005cc4e0)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:206 +0x175
cmd/compile/internal/noder.(*reader2).typ(0xc0005d7680)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:178 +0x38
cmd/compile/internal/noder.(*reader2).structType(0xc0005d7680)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:268 +0xf2
cmd/compile/internal/noder.(*reader2).doTyp(0xc0005d7680)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:254 +0x39c
cmd/compile/internal/noder.(*pkgReader2).typIdx(0xc0000e0900, {0xb0, 0xe0}, 0xc0005cc4e0)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:206 +0x175
cmd/compile/internal/noder.(*reader2).typ(0xc0005cc540)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:178 +0x38
cmd/compile/internal/noder.(*pkgReader2).objIdx.func1.1(0xd1dca0)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:406 +0x3f
cmd/compile/internal/types2.(*Named).load.func1()
	/usr/local/go.tip/src/cmd/compile/internal/types2/named.go:66 +0x2a
sync.(*Once).doSlow(0xb70e27, 0xc0000c80a0)
	/usr/local/go.tip/src/sync/once.go:68 +0xc2
sync.(*Once).Do(...)
	/usr/local/go.tip/src/sync/once.go:59
cmd/compile/internal/types2.(*Named).load(0xc0004addd0)
	/usr/local/go.tip/src/cmd/compile/internal/types2/named.go:56 +0xdb
cmd/compile/internal/types2.(*Named).TParams(...)
	/usr/local/go.tip/src/cmd/compile/internal/types2/named.go:124
cmd/compile/internal/types2.(*typeWriter).typ(0xc0000cff90, {0xe50750, 0xc0004addd0})
	/usr/local/go.tip/src/cmd/compile/internal/types2/typestring.go:237 +0x3bb
cmd/compile/internal/types2.(*typeWriter).typeList(0xc0000cff90, {0xc0003dae80, 0x2, 0x138b1e0})
	/usr/local/go.tip/src/cmd/compile/internal/types2/typestring.go:272 +0xd2
cmd/compile/internal/types2.typeHash({0xe50750, 0xc0004ade60}, {0xc0003dae80, 0x1, 0x2})
	/usr/local/go.tip/src/cmd/compile/internal/types2/subst.go:269 +0xef
cmd/compile/internal/types2.(*Checker).instance(0xc0000f4640, {0x0, 0x0, 0x0}, {0xe50750, 0xc0004ade60}, {0xc0003dae80, 0xc0005a9e90, 0xc0005d70e0})
	/usr/local/go.tip/src/cmd/compile/internal/types2/instantiate.go:128 +0x31f
cmd/compile/internal/types2.Instantiate(0xc0005d70e0, {0xe50750, 0xc0004ade60}, {0xc0003dae80, 0x2, 0x2}, 0x0)
	/usr/local/go.tip/src/cmd/compile/internal/types2/instantiate.go:53 +0xb1
cmd/compile/internal/noder.(*reader2).doTyp(0xc0005d70e0)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:231 +0x150
cmd/compile/internal/noder.(*pkgReader2).typIdx(0xc0000e0900, {0xc0005d7080, 0xb}, 0xc0005d6c60)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:206 +0x175
cmd/compile/internal/noder.(*reader2).typ(0xc0005d7080)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:178 +0x38
cmd/compile/internal/noder.(*reader2).doTyp(0xc0005d7080)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:248 +0x7d
cmd/compile/internal/noder.(*pkgReader2).typIdx(0xc0000e0900, {0xc0005d6cc0, 0x81}, 0xc0005d6c60)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:206 +0x175
cmd/compile/internal/noder.(*reader2).typ(0xc0005d6cc0)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:178 +0x38
cmd/compile/internal/noder.(*reader2).param(0xc0005d6cc0)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:333 +0x69
cmd/compile/internal/noder.(*reader2).params(0xc0005d6cc0)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:323 +0x74
cmd/compile/internal/noder.(*reader2).signature(0xc0005d6cc0, 0x0)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:313 +0x47
cmd/compile/internal/noder.(*pkgReader2).objIdx.func1()
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:392 +0x4f5
cmd/compile/internal/types2.resolve.func1()
	/usr/local/go.tip/src/cmd/compile/internal/types2/scope.go:255 +0x3c
sync.(*Once).doSlow(0x6, 0x7f9623e772cb)
	/usr/local/go.tip/src/sync/once.go:68 +0xc2
sync.(*Once).Do(...)
	/usr/local/go.tip/src/sync/once.go:59
cmd/compile/internal/types2.resolve({0x7f9623e3d55e, 0xc0004fac00}, {0xe61cf8, 0xc0005d24e0})
	/usr/local/go.tip/src/cmd/compile/internal/types2/scope.go:254 +0x85
cmd/compile/internal/types2.(*Scope).Lookup(...)
	/usr/local/go.tip/src/cmd/compile/internal/types2/scope.go:72
cmd/compile/internal/noder.(*reader2).obj(0xc0004ff620)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:346 +0xd0
cmd/compile/internal/noder.readPackage2(0xc0000f4640, 0xc0003d7080, {{0xc0000b24b1, 0x22}, {0x5cc, 0x5cd, 0x607, 0x687, 0x79f, 0xc78, ...}, ...})
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:46 +0x314
cmd/compile/internal/noder.unified.func1({0x7f9623e35ccd, 0x5ccd}, 0xc0005001e0, 0x22, 0x40a3bd)
	/usr/local/go.tip/src/cmd/compile/internal/noder/unified.go:86 +0x108
cmd/compile/internal/noder.readImportFile({0xc0000b24b1, 0xc7f9e0}, 0xc0000e0180, 0xc0003c8901, 0xc0003d7080)
	/usr/local/go.tip/src/cmd/compile/internal/noder/import.go:282 +0x493
cmd/compile/internal/noder.(*gcimports).ImportFrom(0xc77f60, {0xc0000b24b1, 0xc00009f4a0}, {0x0, 0xc0000f18f0}, 0x20)
	/usr/local/go.tip/src/cmd/compile/internal/noder/import.go:64 +0x3a
cmd/compile/internal/types2.(*Checker).importPackage(0xc0000f4640, {0xc000376ea0, 0x0, 0x0}, {0xc0000b24b1, 0x22}, {0x7ffed7a85ba1, 0x9})
	/usr/local/go.tip/src/cmd/compile/internal/types2/resolver.go:144 +0x19e
cmd/compile/internal/types2.(*Checker).collectObjects(0xc0000f4640)
	/usr/local/go.tip/src/cmd/compile/internal/types2/resolver.go:252 +0x10b0
cmd/compile/internal/types2.(*Checker).checkFiles(0xc0000f4640, {0xc0000b45e0, 0x0, 0x0})
	/usr/local/go.tip/src/cmd/compile/internal/types2/check.go:266 +0x1ec
cmd/compile/internal/types2.(*Checker).Files(...)
	/usr/local/go.tip/src/cmd/compile/internal/types2/check.go:245
cmd/compile/internal/noder.checkFiles({0xc0000b4450, 0x1, 0xc0000d1548})
	/usr/local/go.tip/src/cmd/compile/internal/noder/irgen.go:65 +0x485
cmd/compile/internal/noder.writePkgStub({0xc0000b4450, 0x1, 0x1})
	/usr/local/go.tip/src/cmd/compile/internal/noder/unified.go:163 +0x46
cmd/compile/internal/noder.unified({0xc0000b4450, 0x0, 0x2})
	/usr/local/go.tip/src/cmd/compile/internal/noder/unified.go:90 +0x12a
cmd/compile/internal/noder.LoadPackage({0xc0000ba0e0, 0x1, 0x0})
	/usr/local/go.tip/src/cmd/compile/internal/noder/noder.go:84 +0x35d
cmd/compile/internal/gc.Main(0xd1dcb0)
	/usr/local/go.tip/src/cmd/compile/internal/gc/main.go:190 +0xaf3
main.main()
	/usr/local/go.tip/src/cmd/compile/main.go:55 +0xdd
FAIL	github.com/go-json-experiment/json [build failed]

The relevant change that seems to cause an ICE is the code that instantiates a generic type:

Unmarshalers: jsonv2.UnmarshalFuncV2(func(opts jsonv2.UnmarshalOptions, dec *jsonv2.Decoder, val *interface{}) (err error) {
	if dec.PeekKind() == '0' {
		*val = jsonv2.RawValue(nil)
	}
	return jsonv2.SkipFunc
}),

\cc @danscales @mdempsky @griesemer @ianlancetaylor

[cuonglm]

I can't reproduce this with:

go version devel go1.18-4957976b1a Fri Sep 3 22:56:58 2021 +0000 darwin/arm64

$ go test
PASS
ok  	github.com/go-json-experiment/json	0.813s

$ GOEXPERIMENT=unified go test
PASS
ok  	github.com/go-json-experiment/json	0.834s

[dsnet]

What branch are you on?

[cuonglm]

What branch are you on?

Oh, ops, sorry, need to checkout origin/generics-bug4

[anacrolix]

I get the same error with anacrolix/stm@80e033a, running go test ./rate.

# github.com/anacrolix/stm/rate [github.com/anacrolix/stm/rate.test]
<autogenerated>:1: internal compiler error: Dictionary should have already been generated: github.com/anacrolix/stm..dict.(*Var[bool]).Set

goroutine 1 [running]:
runtime/debug.Stack()
	/Users/anacrolix/src/go.master/src/runtime/debug/stack.go:24 +0x65
cmd/compile/internal/base.FatalfAt({0x27faeca, 0x0}, {0x190f2b3, 0x34}, {0xc0004caec0, 0x2, 0x2})
	/Users/anacrolix/src/go.master/src/cmd/compile/internal/base/print.go:227 +0x154
cmd/compile/internal/base.Fatalf(...)
	/Users/anacrolix/src/go.master/src/cmd/compile/internal/base/print.go:196
cmd/compile/internal/reflectdata.getDictionary(0xc000961630, {0xc000a1a4b8, 0x1, 0x1})
	/Users/anacrolix/src/go.master/src/cmd/compile/internal/reflectdata/reflect.go:2043 +0x245
cmd/compile/internal/reflectdata.methodWrapper(0xc0008a79d0, 0xc000961680, 0x40)
	/Users/anacrolix/src/go.master/src/cmd/compile/internal/reflectdata/reflect.go:1909 +0x10ea
cmd/compile/internal/reflectdata.methods(0xc0008a79d0)
	/Users/anacrolix/src/go.master/src/cmd/compile/internal/reflectdata/reflect.go:347 +0x39f
cmd/compile/internal/reflectdata.writeITab(0xc000b27e00, 0xc0008a79d0, 0xc00086a070)
	/Users/anacrolix/src/go.master/src/cmd/compile/internal/reflectdata/reflect.go:1290 +0x26a
cmd/compile/internal/reflectdata.ITabLsym(0xc00086a070, 0xc0008776c0)
	/Users/anacrolix/src/go.master/src/cmd/compile/internal/reflectdata/reflect.go:847 +0x19c
cmd/compile/internal/noder.(*irgen).finalizeSyms(0xc0000bae00)
	/Users/anacrolix/src/go.master/src/cmd/compile/internal/noder/stencil.go:1632 +0x7a5
cmd/compile/internal/noder.(*irgen).stencil(0xc0000bae00)
	/Users/anacrolix/src/go.master/src/cmd/compile/internal/noder/stencil.go:234 +0x46d
cmd/compile/internal/noder.(*irgen).generate(0xc0000bae00, {0xc000096b20, 0x2, 0xc0})
	/Users/anacrolix/src/go.master/src/cmd/compile/internal/noder/irgen.go:304 +0x359
cmd/compile/internal/noder.check2({0xc000096b20, 0x2, 0x2})
	/Users/anacrolix/src/go.master/src/cmd/compile/internal/noder/irgen.go:92 +0x175
cmd/compile/internal/noder.LoadPackage({0xc0000ba0e0, 0x2, 0x0})
	/Users/anacrolix/src/go.master/src/cmd/compile/internal/noder/noder.go:90 +0x335
cmd/compile/internal/gc.Main(0x191aec0)
	/Users/anacrolix/src/go.master/src/cmd/compile/internal/gc/main.go:190 +0xaf3
main.main()
	/Users/anacrolix/src/go.master/src/cmd/compile/main.go:55 +0xdd

FAIL	github.com/anacrolix/stm/rate [build failed]
FAIL

[cuonglm]

Simpler reproducer for unified IR:

$ cat p.go
package p

type MarshalOptions struct {
	Marshalers *Marshalers
}

type Encoder struct {}

type Marshalers = arshalers[MarshalOptions, Encoder]

type arshalers[Options, Coder any] struct{}

func MarshalFuncV1[T any](fn func(T) ([]byte, error)) *Marshalers {
	return &Marshalers{}
}
$ cat p_test.go
package p

I think this is problem of types2 lazy loading API. In (*Named).load, we call t.resolve(t) which may refer back to the same named type, so calling (*Named).load again, causing the deadlock. In above code, it's the arshalers type. I can't find a way to trigger the bug purely in types2, though.

cc @mdempsky @griesemer

[danscales]

I have a fix for the -G=3 version of the issue. I will not close automatically, so the bug can remain open for the unified IR version of the issue. Or we can create separate issues.

[gopherbot]

Change https://golang.org/cl/347909 mentions this issue: cmd/compile: make sure imported instantiated types have their methods created

[danscales]

Thanks for all the issue reports, @dsnet and @anacrolix . They are all useful, so don't worry about filing them. Of course, somewhat minimized cases are always appreciated (so it is easier to add a test case).

[gopherbot]

Change https://golang.org/cl/347534 mentions this issue: cmd/compile: fix deadlock in (*Named).load