Skip to content

cmd/compile: internal compiler error instantiating generic type #48185

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
dsnet opened this issue Sep 3, 2021 · 9 comments
Closed

cmd/compile: internal compiler error instantiating generic type #48185

dsnet opened this issue Sep 3, 2021 · 9 comments
Labels
FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Milestone
Go1.18

Comments

@dsnet
Copy link
Member

dsnet commented Sep 3, 2021

I apologize for all the bug reports, but I'm hitting another internal compiler error.

$ go.tip version
go version devel go1.18-d9244f8b64 Fri Sep 3 21:31:30 2021 +0000 linux/amd64

$ go.tip test
# github.com/go-json-experiment/json_test [github.com/go-json-experiment/json.test]
<autogenerated>:1: internal compiler error: Dictionary should have already been generated: github.com/go-json-experiment/json..dict.(*arshalers[json.UnmarshalOptions,json.Decoder]).lookup

goroutine 1 [running]:
runtime/debug.Stack()
	/usr/local/go.tip/src/runtime/debug/stack.go:24 +0x65
cmd/compile/internal/base.FatalfAt({0x9df580, 0xc0}, {0xd120ee, 0x34}, {0xc000589320, 0x2, 0x2})
	/usr/local/go.tip/src/cmd/compile/internal/base/print.go:227 +0x154
cmd/compile/internal/base.Fatalf(...)
	/usr/local/go.tip/src/cmd/compile/internal/base/print.go:196
cmd/compile/internal/reflectdata.getDictionary(0xc000732eb0, {0xc0006b1c00, 0x2, 0x2})
	/usr/local/go.tip/src/cmd/compile/internal/reflectdata/reflect.go:2043 +0x245
cmd/compile/internal/reflectdata.methodWrapper(0xc000730070, 0xc000732f00, 0xf5)
	/usr/local/go.tip/src/cmd/compile/internal/reflectdata/reflect.go:1909 +0x10ea
cmd/compile/internal/reflectdata.methods(0xc000730070)
	/usr/local/go.tip/src/cmd/compile/internal/reflectdata/reflect.go:347 +0x39f
cmd/compile/internal/reflectdata.uncommonSize(...)
	/usr/local/go.tip/src/cmd/compile/internal/reflectdata/reflect.go:79
cmd/compile/internal/reflectdata.dcommontype(0xc0006dbf80, 0xc000730070)
	/usr/local/go.tip/src/cmd/compile/internal/reflectdata/reflect.go:716 +0x1cc
cmd/compile/internal/reflectdata.writeType(0xc0006b29a0)
	/usr/local/go.tip/src/cmd/compile/internal/reflectdata/reflect.go:1140 +0x327
cmd/compile/internal/reflectdata.WriteRuntimeTypes()
	/usr/local/go.tip/src/cmd/compile/internal/reflectdata/reflect.go:1254 +0x3d8
cmd/compile/internal/gc.dumpdata()
	/usr/local/go.tip/src/cmd/compile/internal/gc/obj.go:118 +0x6e
cmd/compile/internal/gc.Main(0xd1dcb0)
	/usr/local/go.tip/src/cmd/compile/internal/gc/main.go:327 +0x10de
main.main()
	/usr/local/go.tip/src/cmd/compile/main.go:55 +0xdd

FAIL	github.com/go-json-experiment/json [build failed]

Reproduction:

$ git clone https://github.com/go-json-experiment/json.git
$ cd json
$ git checkout origin/generics-bug4
$ go test

Based on @mdempsky's comment in #48016 (comment), I tried building this with GOEXPERIMENT=unified and that also panics, but due to a different problem:

$ GOEXPERIMENT=unified go.tip test
# github.com/go-json-experiment/json_test [github.com/go-json-experiment/json.test]
fatal error: all goroutines are asleep - deadlock!

goroutine 1 [semacquire]:
sync.runtime_SemacquireMutex(0xc0000cf068, 0x98, 0xa2)
	/usr/local/go.tip/src/runtime/sema.go:71 +0x25
sync.(*Mutex).lockSlow(0xc0004ade4c)
	/usr/local/go.tip/src/sync/mutex.go:138 +0x165
sync.(*Mutex).Lock(...)
	/usr/local/go.tip/src/sync/mutex.go:81
sync.(*Once).doSlow(0xb70e27, 0xc0000cf0e0)
	/usr/local/go.tip/src/sync/once.go:64 +0x53
sync.(*Once).Do(...)
	/usr/local/go.tip/src/sync/once.go:59
cmd/compile/internal/types2.(*Named).load(0xc0004addd0)
	/usr/local/go.tip/src/cmd/compile/internal/types2/named.go:56 +0xdb
cmd/compile/internal/types2.(*Named).TParams(...)
	/usr/local/go.tip/src/cmd/compile/internal/types2/named.go:124
cmd/compile/internal/types2.(*typeWriter).typ(0xc0000cf398, {0xe50750, 0xc0004addd0})
	/usr/local/go.tip/src/cmd/compile/internal/types2/typestring.go:237 +0x3bb
cmd/compile/internal/types2.(*typeWriter).typeList(0xc0000cf398, {0xc0003daec0, 0x2, 0x138b1e0})
	/usr/local/go.tip/src/cmd/compile/internal/types2/typestring.go:272 +0xd2
cmd/compile/internal/types2.typeHash({0xe50750, 0xc0004ade60}, {0xc0003daec0, 0x10, 0x2})
	/usr/local/go.tip/src/cmd/compile/internal/types2/subst.go:269 +0xef
cmd/compile/internal/types2.(*Checker).instance(0xc0000f4640, {0x0, 0x0, 0x0}, {0xe50750, 0xc0004ade60}, {0xc0003daec0, 0xc0005a9e90, 0xc0005d7bc0})
	/usr/local/go.tip/src/cmd/compile/internal/types2/instantiate.go:128 +0x31f
cmd/compile/internal/types2.Instantiate(0xc0005d7bc0, {0xe50750, 0xc0004ade60}, {0xc0003daec0, 0x2, 0x2}, 0x0)
	/usr/local/go.tip/src/cmd/compile/internal/types2/instantiate.go:53 +0xb1
cmd/compile/internal/noder.(*reader2).doTyp(0xc0005d7bc0)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:231 +0x150
cmd/compile/internal/noder.(*pkgReader2).typIdx(0xc0000e0900, {0xc0005d7b60, 0xb}, 0xc0005cc4e0)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:206 +0x175
cmd/compile/internal/noder.(*reader2).typ(0xc0005d7b60)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:178 +0x38
cmd/compile/internal/noder.(*reader2).doTyp(0xc0005d7b60)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:248 +0x7d
cmd/compile/internal/noder.(*pkgReader2).typIdx(0xc0000e0900, {0xc0005d7680, 0x9}, 0xc0005cc4e0)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:206 +0x175
cmd/compile/internal/noder.(*reader2).typ(0xc0005d7680)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:178 +0x38
cmd/compile/internal/noder.(*reader2).structType(0xc0005d7680)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:268 +0xf2
cmd/compile/internal/noder.(*reader2).doTyp(0xc0005d7680)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:254 +0x39c
cmd/compile/internal/noder.(*pkgReader2).typIdx(0xc0000e0900, {0xb0, 0xe0}, 0xc0005cc4e0)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:206 +0x175
cmd/compile/internal/noder.(*reader2).typ(0xc0005cc540)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:178 +0x38
cmd/compile/internal/noder.(*pkgReader2).objIdx.func1.1(0xd1dca0)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:406 +0x3f
cmd/compile/internal/types2.(*Named).load.func1()
	/usr/local/go.tip/src/cmd/compile/internal/types2/named.go:66 +0x2a
sync.(*Once).doSlow(0xb70e27, 0xc0000c80a0)
	/usr/local/go.tip/src/sync/once.go:68 +0xc2
sync.(*Once).Do(...)
	/usr/local/go.tip/src/sync/once.go:59
cmd/compile/internal/types2.(*Named).load(0xc0004addd0)
	/usr/local/go.tip/src/cmd/compile/internal/types2/named.go:56 +0xdb
cmd/compile/internal/types2.(*Named).TParams(...)
	/usr/local/go.tip/src/cmd/compile/internal/types2/named.go:124
cmd/compile/internal/types2.(*typeWriter).typ(0xc0000cff90, {0xe50750, 0xc0004addd0})
	/usr/local/go.tip/src/cmd/compile/internal/types2/typestring.go:237 +0x3bb
cmd/compile/internal/types2.(*typeWriter).typeList(0xc0000cff90, {0xc0003dae80, 0x2, 0x138b1e0})
	/usr/local/go.tip/src/cmd/compile/internal/types2/typestring.go:272 +0xd2
cmd/compile/internal/types2.typeHash({0xe50750, 0xc0004ade60}, {0xc0003dae80, 0x1, 0x2})
	/usr/local/go.tip/src/cmd/compile/internal/types2/subst.go:269 +0xef
cmd/compile/internal/types2.(*Checker).instance(0xc0000f4640, {0x0, 0x0, 0x0}, {0xe50750, 0xc0004ade60}, {0xc0003dae80, 0xc0005a9e90, 0xc0005d70e0})
	/usr/local/go.tip/src/cmd/compile/internal/types2/instantiate.go:128 +0x31f
cmd/compile/internal/types2.Instantiate(0xc0005d70e0, {0xe50750, 0xc0004ade60}, {0xc0003dae80, 0x2, 0x2}, 0x0)
	/usr/local/go.tip/src/cmd/compile/internal/types2/instantiate.go:53 +0xb1
cmd/compile/internal/noder.(*reader2).doTyp(0xc0005d70e0)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:231 +0x150
cmd/compile/internal/noder.(*pkgReader2).typIdx(0xc0000e0900, {0xc0005d7080, 0xb}, 0xc0005d6c60)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:206 +0x175
cmd/compile/internal/noder.(*reader2).typ(0xc0005d7080)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:178 +0x38
cmd/compile/internal/noder.(*reader2).doTyp(0xc0005d7080)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:248 +0x7d
cmd/compile/internal/noder.(*pkgReader2).typIdx(0xc0000e0900, {0xc0005d6cc0, 0x81}, 0xc0005d6c60)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:206 +0x175
cmd/compile/internal/noder.(*reader2).typ(0xc0005d6cc0)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:178 +0x38
cmd/compile/internal/noder.(*reader2).param(0xc0005d6cc0)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:333 +0x69
cmd/compile/internal/noder.(*reader2).params(0xc0005d6cc0)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:323 +0x74
cmd/compile/internal/noder.(*reader2).signature(0xc0005d6cc0, 0x0)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:313 +0x47
cmd/compile/internal/noder.(*pkgReader2).objIdx.func1()
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:392 +0x4f5
cmd/compile/internal/types2.resolve.func1()
	/usr/local/go.tip/src/cmd/compile/internal/types2/scope.go:255 +0x3c
sync.(*Once).doSlow(0x6, 0x7f9623e772cb)
	/usr/local/go.tip/src/sync/once.go:68 +0xc2
sync.(*Once).Do(...)
	/usr/local/go.tip/src/sync/once.go:59
cmd/compile/internal/types2.resolve({0x7f9623e3d55e, 0xc0004fac00}, {0xe61cf8, 0xc0005d24e0})
	/usr/local/go.tip/src/cmd/compile/internal/types2/scope.go:254 +0x85
cmd/compile/internal/types2.(*Scope).Lookup(...)
	/usr/local/go.tip/src/cmd/compile/internal/types2/scope.go:72
cmd/compile/internal/noder.(*reader2).obj(0xc0004ff620)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:346 +0xd0
cmd/compile/internal/noder.readPackage2(0xc0000f4640, 0xc0003d7080, {{0xc0000b24b1, 0x22}, {0x5cc, 0x5cd, 0x607, 0x687, 0x79f, 0xc78, ...}, ...})
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:46 +0x314
cmd/compile/internal/noder.unified.func1({0x7f9623e35ccd, 0x5ccd}, 0xc0005001e0, 0x22, 0x40a3bd)
	/usr/local/go.tip/src/cmd/compile/internal/noder/unified.go:86 +0x108
cmd/compile/internal/noder.readImportFile({0xc0000b24b1, 0xc7f9e0}, 0xc0000e0180, 0xc0003c8901, 0xc0003d7080)
	/usr/local/go.tip/src/cmd/compile/internal/noder/import.go:282 +0x493
cmd/compile/internal/noder.(*gcimports).ImportFrom(0xc77f60, {0xc0000b24b1, 0xc00009f4a0}, {0x0, 0xc0000f18f0}, 0x20)
	/usr/local/go.tip/src/cmd/compile/internal/noder/import.go:64 +0x3a
cmd/compile/internal/types2.(*Checker).importPackage(0xc0000f4640, {0xc000376ea0, 0x0, 0x0}, {0xc0000b24b1, 0x22}, {0x7ffed7a85ba1, 0x9})
	/usr/local/go.tip/src/cmd/compile/internal/types2/resolver.go:144 +0x19e
cmd/compile/internal/types2.(*Checker).collectObjects(0xc0000f4640)
	/usr/local/go.tip/src/cmd/compile/internal/types2/resolver.go:252 +0x10b0
cmd/compile/internal/types2.(*Checker).checkFiles(0xc0000f4640, {0xc0000b45e0, 0x0, 0x0})
	/usr/local/go.tip/src/cmd/compile/internal/types2/check.go:266 +0x1ec
cmd/compile/internal/types2.(*Checker).Files(...)
	/usr/local/go.tip/src/cmd/compile/internal/types2/check.go:245
cmd/compile/internal/noder.checkFiles({0xc0000b4450, 0x1, 0xc0000d1548})
	/usr/local/go.tip/src/cmd/compile/internal/noder/irgen.go:65 +0x485
cmd/compile/internal/noder.writePkgStub({0xc0000b4450, 0x1, 0x1})
	/usr/local/go.tip/src/cmd/compile/internal/noder/unified.go:163 +0x46
cmd/compile/internal/noder.unified({0xc0000b4450, 0x0, 0x2})
	/usr/local/go.tip/src/cmd/compile/internal/noder/unified.go:90 +0x12a
cmd/compile/internal/noder.LoadPackage({0xc0000ba0e0, 0x1, 0x0})
	/usr/local/go.tip/src/cmd/compile/internal/noder/noder.go:84 +0x35d
cmd/compile/internal/gc.Main(0xd1dcb0)
	/usr/local/go.tip/src/cmd/compile/internal/gc/main.go:190 +0xaf3
main.main()
	/usr/local/go.tip/src/cmd/compile/main.go:55 +0xdd
FAIL	github.com/go-json-experiment/json [build failed]

The relevant change that seems to cause an ICE is the code that instantiates a generic type:

Unmarshalers: jsonv2.UnmarshalFuncV2(func(opts jsonv2.UnmarshalOptions, dec *jsonv2.Decoder, val *interface{}) (err error) {
	if dec.PeekKind() == '0' {
		*val = jsonv2.RawValue(nil)
	}
	return jsonv2.SkipFunc
}),

\cc @danscales @mdempsky @griesemer @ianlancetaylor
@cherrymui cherrymui added the NeedsFix The path to resolution is known, but the work has not been done. label Sep 3, 2021
@cherrymui cherrymui added this to the Go1.18 milestone Sep 3, 2021
@cuonglm

This comment has been minimized.

Sign in to view
@dsnet

This comment has been minimized.

Sign in to view
@cuonglm

This comment has been minimized.

Sign in to view
@anacrolix
Copy link
Contributor

I get the same error with anacrolix/stm@80e033a, running go test ./rate.

# github.com/anacrolix/stm/rate [github.com/anacrolix/stm/rate.test]
<autogenerated>:1: internal compiler error: Dictionary should have already been generated: github.com/anacrolix/stm..dict.(*Var[bool]).Set

goroutine 1 [running]:
runtime/debug.Stack()
	/Users/anacrolix/src/go.master/src/runtime/debug/stack.go:24 +0x65
cmd/compile/internal/base.FatalfAt({0x27faeca, 0x0}, {0x190f2b3, 0x34}, {0xc0004caec0, 0x2, 0x2})
	/Users/anacrolix/src/go.master/src/cmd/compile/internal/base/print.go:227 +0x154
cmd/compile/internal/base.Fatalf(...)
	/Users/anacrolix/src/go.master/src/cmd/compile/internal/base/print.go:196
cmd/compile/internal/reflectdata.getDictionary(0xc000961630, {0xc000a1a4b8, 0x1, 0x1})
	/Users/anacrolix/src/go.master/src/cmd/compile/internal/reflectdata/reflect.go:2043 +0x245
cmd/compile/internal/reflectdata.methodWrapper(0xc0008a79d0, 0xc000961680, 0x40)
	/Users/anacrolix/src/go.master/src/cmd/compile/internal/reflectdata/reflect.go:1909 +0x10ea
cmd/compile/internal/reflectdata.methods(0xc0008a79d0)
	/Users/anacrolix/src/go.master/src/cmd/compile/internal/reflectdata/reflect.go:347 +0x39f
cmd/compile/internal/reflectdata.writeITab(0xc000b27e00, 0xc0008a79d0, 0xc00086a070)
	/Users/anacrolix/src/go.master/src/cmd/compile/internal/reflectdata/reflect.go:1290 +0x26a
cmd/compile/internal/reflectdata.ITabLsym(0xc00086a070, 0xc0008776c0)
	/Users/anacrolix/src/go.master/src/cmd/compile/internal/reflectdata/reflect.go:847 +0x19c
cmd/compile/internal/noder.(*irgen).finalizeSyms(0xc0000bae00)
	/Users/anacrolix/src/go.master/src/cmd/compile/internal/noder/stencil.go:1632 +0x7a5
cmd/compile/internal/noder.(*irgen).stencil(0xc0000bae00)
	/Users/anacrolix/src/go.master/src/cmd/compile/internal/noder/stencil.go:234 +0x46d
cmd/compile/internal/noder.(*irgen).generate(0xc0000bae00, {0xc000096b20, 0x2, 0xc0})
	/Users/anacrolix/src/go.master/src/cmd/compile/internal/noder/irgen.go:304 +0x359
cmd/compile/internal/noder.check2({0xc000096b20, 0x2, 0x2})
	/Users/anacrolix/src/go.master/src/cmd/compile/internal/noder/irgen.go:92 +0x175
cmd/compile/internal/noder.LoadPackage({0xc0000ba0e0, 0x2, 0x0})
	/Users/anacrolix/src/go.master/src/cmd/compile/internal/noder/noder.go:90 +0x335
cmd/compile/internal/gc.Main(0x191aec0)
	/Users/anacrolix/src/go.master/src/cmd/compile/internal/gc/main.go:190 +0xaf3
main.main()
	/Users/anacrolix/src/go.master/src/cmd/compile/main.go:55 +0xdd

FAIL	github.com/anacrolix/stm/rate [build failed]
FAIL

@cuonglm
Copy link
Member

cuonglm commented Sep 5, 2021 •
edited
Loading

Simpler reproducer for unified IR:

$ cat p.go
package p

type MarshalOptions struct {
	Marshalers *Marshalers
}

type Encoder struct {}

type Marshalers = arshalers[MarshalOptions, Encoder]

type arshalers[Options, Coder any] struct{}

func MarshalFuncV1[T any](fn func(T) ([]byte, error)) *Marshalers {
	return &Marshalers{}
}
$ cat p_test.go
package p

I think this is problem of types2 lazy loading API. In (*Named).load, we call t.resolve(t) which may refer back to the same named type, so calling (*Named).load again, causing the deadlock. In above code, it's the arshalers type. I can't find a way to trigger the bug purely in types2, though.

cc @mdempsky @griesemer

@danscales
Copy link
Contributor

I have a fix for the -G=3 version of the issue. I will not close automatically, so the bug can remain open for the unified IR version of the issue. Or we can create separate issues.

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/347909 mentions this issue: cmd/compile: make sure imported instantiated types have their methods created

@danscales
Copy link
Contributor

Thanks for all the issue reports, @dsnet and @anacrolix . They are all useful, so don't worry about filing them. Of course, somewhat minimized cases are always appreciated (so it is easier to add a test case).

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/347534 mentions this issue: cmd/compile: fix deadlock in (*Named).load

gopherbot pushed a commit that referenced this issue Sep 7, 2021
@cuonglm
cmd/compile: fix deadlock in (*Named).load
2a48452 
For lazy import resolution, there's reentrancy issue with (*Named).load
method, when "t.resolve(t)" can lead us to the same named type, thus
(*Named).load is called recursively, causing the deadlock.

The main problem is that when instantinate a type, we calculate the type
hashing, including TParams. Calling t.TParams().Len() triggers the
reentrancy call to "(*Named).load".

To fix this, just not checking TParams().Len() if we are hashing.

Updates #48185

Change-Id: Ie34842d7b10fad5d11fbcf75bb1c64a89deac6b8
Reviewed-on: https://go-review.googlesource.com/c/go/+/347534
Trust: Cuong Manh Le <cuong.manhle.vn@gmail.com>
Trust: Robert Griesemer <gri@golang.org>
Run-TryBot: Cuong Manh Le <cuong.manhle.vn@gmail.com>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
Reviewed-by: Robert Findley <rfindley@google.com>
Reviewed-by: Robert Griesemer <gri@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
@ALTree ALTree mentioned this issue Jan 12, 2022
Closed
@golang golang locked and limited conversation to collaborators Sep 7, 2022
@julieqiu julieqiu removed this from Go Security Sep 8, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Milestone
Go1.18
Development

No branches or pull requests
6 participants
@anacrolix @danscales @cuonglm @dsnet @gopherbot @cherrymui