cue/cmd: support generic Go types in `cue get go` #2217

rogpeppe · 2023-01-16T18:11:43Z

Currently, use of generic structs results in a panic.

For example, this example:

exec cue get go .
cmp cue.mod/pkg/test/foo.cue expect-foo.cue

-- cue.mod/module.cue --
module: "example.com"

-- go.mod --
module test

-- foo.go --
package foo

type S[T C] struct {
	X T
}

type C interface {
	int | string
}
-- foo.cue --
package foo

#S: {
	X: #C
}

#C: int | string

results in the following testscript failure:

> exec cue get go .
[stderr]
panic: unsupported type *types.TypeParam [recovered]
	panic: unsupported type *types.TypeParam

goroutine 1 [running]:
cuelang.org/go/cmd/cue/cmd.recoverError(0xc0003a5e90)
	/home/rogpeppe/src/cuelabs/cue2/cmd/cue/cmd/root.go:232 +0x7b
panic({0xb559a0, 0xc000247e90})
	/home/rogpeppe/go/src/runtime/panic.go:884 +0x212
cuelang.org/go/cmd/cue/cmd.(*extractor).makeType(0xc0005a4480, {0xd7a360?, 0xc000418a50?})
	/home/rogpeppe/src/cuelabs/cue2/cmd/cue/cmd/get_go.go:1139 +0x107d
cuelang.org/go/cmd/cue/cmd.(*extractor).makeField(0x119d8a0?, {0x119d8a0, 0x1}, 0x2f, {0xd7a360?, 0xc000418a50?}, 0xc000224000?, 0x0?)
	/home/rogpeppe/src/cuelabs/cue2/cmd/cue/cmd/get_go.go:987 +0x6e
cuelang.org/go/cmd/cue/cmd.(*extractor).addFields(0xc0005a4480, 0xc000418b70, 0xc000099ae0)
	/home/rogpeppe/src/cuelabs/cue2/cmd/cue/cmd/get_go.go:1212 +0x3ab
cuelang.org/go/cmd/cue/cmd.(*extractor).makeType(0xc0005a4480, {0xd7a310?, 0xc000418b70?})
	/home/rogpeppe/src/cuelabs/cue2/cmd/cue/cmd/get_go.go:1078 +0xfb9
cuelang.org/go/cmd/cue/cmd.(*extractor).makeField(0x0?, {0x119d878, 0x1}, 0x30, {0xd7a310?, 0xc000418b70?}, 0x44df45?, 0x50?)
	/home/rogpeppe/src/cuelabs/cue2/cmd/cue/cmd/get_go.go:987 +0x6e
cuelang.org/go/cmd/cue/cmd.(*extractor).reportDecl(0xc0005a4480, 0xc0004ce2c0)
	/home/rogpeppe/src/cuelabs/cue2/cmd/cue/cmd/get_go.go:699 +0x445
cuelang.org/go/cmd/cue/cmd.(*extractor).extractPkg(0xc0005a4480, {0xc000038094, 0x2a}, 0xc0005ae000)
	/home/rogpeppe/src/cuelabs/cue2/cmd/cue/cmd/get_go.go:501 +0xe5c
cuelang.org/go/cmd/cue/cmd.extract(0xc0003549c0, {0xc000247ed0, 0x1, 0x1})
	/home/rogpeppe/src/cuelabs/cue2/cmd/cue/cmd/get_go.go:417 +0x3e5
cuelang.org/go/cmd/cue/cmd.mkRunE.func1(0xc000374500?, {0xc000247ed0?, 0x1?, 0x1?})
	/home/rogpeppe/src/cuelabs/cue2/cmd/cue/cmd/root.go:48 +0x4d
github.com/spf13/cobra.(*Command).execute(0xc000374500, {0xc000247ea0, 0x1, 0x1})
	/home/rogpeppe/src/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:856 +0x67c
github.com/spf13/cobra.(*Command).ExecuteC(0xc00035ec80)
	/home/rogpetestscript: exit 1
ppe/src/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:974 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
	/home/rogpeppe/src/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:902
cuelang.org/go/cmd/cue/cmd.(*Command).Run(0xc0003549c0, {0x3?, 0x3?})
	/home/rogpeppe/src/cuelabs/cue2/cmd/cue/cmd/root.go:217 +0x6b
cuelang.org/go/cmd/cue/cmd.mainErr({0xd7d5b0, 0xc00003c0a0}, {0xc0000360d0?, 0xc0000061a0?, 0x200000003?})
	/home/rogpeppe/src/cuelabs/cue2/cmd/cue/cmd/root.go:154 +0x55
cuelang.org/go/cmd/cue/cmd.Main()
	/home/rogpeppe/src/cuelabs/cue2/cmd/cue/cmd/root.go:136 +0x7c
main.main()
	/home/rogpeppe/src/cuelabs/cue2/cmd/cue/main.go:24 +0x19
[exit status 2]
FAIL: /tmp/testscript610380257/x.txtar/script.txtar:1: unexpected command failure

It would be better if it generated reasonable code similar to what's expected above instead.
r context or screenshots about the feature request here.

The text was updated successfully, but these errors were encountered:

mvdan · 2023-01-17T16:17:05Z

This feels somewhat urgent, even if we choose to treat generic types in a lazy way like "top", because generics are becoming increasingly common in Go code. And cue get go stops being helpful once it runs into panics like these.

myitcv · 2023-02-02T16:39:37Z

@rogpeppe

What version of CUE was used?
What version of Go was used?

uhthomas · 2023-05-26T23:48:58Z

Latest for both (v0.5.0) - I just encountered the same issue.

myitcv · 2023-06-14T11:49:36Z

Will revisit when we plan/schedule the feature requests.

verdverm · 2023-08-12T23:25:19Z

I'm hitting this error when trying to cue get go any recent version of the k8s schemas

verdverm · 2023-08-12T23:34:32Z

reproducer, I see a change in errors for these k8s version, but changing go/cue version does not matter

note, Go 1.21 will not work because cue was compiled with 1.20

go mod init hof.io/repro
go mod tidy
go run dagger.go --k8s 0.25.12 --go 1.20 --cue 0.6.0     # panics about invalid memory address or nil pointer dereference
go run dagger.go --k8s 0.26.7 --go 1.20 --cue 0.6.0       # panics about *types.TypeParam

-- dagger.go --
package main

import (
	"context"
	"fmt"
	"os"

	"dagger.io/dagger"
	"github.com/spf13/pflag"
)

func check(err error) {
	if err != nil {
		fmt.Println(err)
		os.Exit(1)
	}
}

var K8S_VERSION string
var CUE_VERSION string
var HOF_VERSION string
var GO_VERSION string

func init() {
	pflag.StringVar(&K8S_VERSION, "k8s", "0.27.4", "k8s.io/api module version to use, see https://github.com/kubernetes/client-go/tags for available versions")
	pflag.StringVar(&CUE_VERSION, "cue", "0.6.0", "CUE/cue version to use, see https://github.com/cue-lang/cue/tags for available versions")
	pflag.StringVar(&HOF_VERSION, "hof", "0.6.8", "hof version to use, see https://github.com/hofstadter-io/hof/tags for available versions")
	pflag.StringVar(&GO_VERSION, "go", "1.20.7", "Go version to use, cannot be newer than CUE or k8s uses")
}

func main() {
	pflag.Parse()
	ctx := context.Background()

	// initialize Dagger client
	client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stdout))
	check(err)
	defer client.Close()

	c, err := baseImage(client).
		With(printVersions).
		With(fetchDeps).
		With(fetchCode).
		Sync(ctx)
	check(err)

	// do things
	c, err = c.With(finalize).Sync(ctx)
}

func baseImage(client *dagger.Client) (*dagger.Container) {
	// our base image
	golang := client.Container().
		From("golang:"+GO_VERSION).
		WithWorkdir("/work")

	// the container we will build up
	c := golang.Pipeline("base")
		
	// go mod cache between runs
	modCache := client.CacheVolume("gomod-global")
	c = c.WithMountedCache("/go/pkg/mod", modCache)

	// extra packages
	c = c.WithExec([]string{
		"apt-get", "update", "-y",
	})
	c = c.WithExec([]string{
		"apt-get", "install", "-y", 
		"gcc",
		"git",
		"make",
		"python3",
		"tar",
		"tree",
		"wget",
	})

	// CUE binary, fetched in a different container (think of this like a multi-stage Dockerfile)
	url := fmt.Sprintf("https://github.com/cue-lang/cue/releases/download/v%s/cue_v%s_linux_amd64.tar.gz", CUE_VERSION, CUE_VERSION)
	tar := fmt.Sprintf("cue.tar.gz")
	cue := golang.Pipeline("cue").
		WithExec([]string{ "wget", url, "-O", tar}).
		WithExec([]string{ "tar", "-xf", tar}).
		WithExec([]string{ "chmod", "+x", "cue"}).
		File("/work/cue")

	// add CUE binary to our container
	c = c.WithFile("/usr/local/bin/cue", cue)

	// hof binary, fetched in a different container (think of this like a multi-stage Dockerfile)
	url = fmt.Sprintf("https://github.com/hofstadter-io/hof/releases/download/v%s/hof_v%s_Linux_x86_64", HOF_VERSION, HOF_VERSION)
	hof := golang.Pipeline("hof").
		WithExec([]string{ "wget", url, "-O", "hof"}).
		WithExec([]string{ "chmod", "+x", "hof"}).
		File("/work/hof")

	// add hof binary to our container
	c = c.WithFile("/usr/local/bin/hof", hof)

	return c
}

func fetchDeps(c *dagger.Container) (*dagger.Container) {
	return c.Pipeline("deps").
		WithExec([]string{"go", "mod", "init", "hof.io/hack"}).
		WithExec([]string{"go", "get", "k8s.io/api@v" + K8S_VERSION}).
		WithExec([]string{"go", "get", "k8s.io/apimachinery@v" + K8S_VERSION})
}

func fetchCode(c *dagger.Container) (*dagger.Container) {
	return c.Pipeline("import").
		WithExec([]string{"cue", "get", "go", "k8s.io/api/..."}).
		WithExec([]string{"cue", "get", "go", "k8s.io/apimachinery/..."})
}

func printVersions(c *dagger.Container) (*dagger.Container) {
	return c.Pipeline("versions").
		WithExec([]string{"go", "version"}).
		WithExec([]string{"cue", "version"})
		// WithExec([]string{"hof", "version"})
}

func finalize(c *dagger.Container) (*dagger.Container) {
	return c.Pipeline("finalize").
		WithExec([]string{"tree"})
}

verdverm · 2023-08-13T00:53:42Z

Interesting, if I change the apimachinery line to the following, it works across many versions of k8s

		WithExec([]string{"cue", "get", "go", "k8s.io/apimachinery/pkg/api/..."})

myitcv · 2023-08-17T05:53:27Z

@verdverm thanks for the report. Just to summarise my understanding of the various issues at play (I couldn't get your repro to run for me).

Given:

# repro.txtar

go install golang.org/dl/$GO_VERSION@latest
go get cuelang.org/go@$CUE_VERSION
go get k8s.io/apimachinery@$K8S_VERSION
go get k8s.io/api@$K8S_VERSION
go mod tidy
exec cat go.mod

go run cuelang.org/go/cmd/cue get go k8s.io/api/...
go run cuelang.org/go/cmd/cue get go $MACHINERY

-- cue.mod/module.cue --
module: "example.com"
-- go.mod --
module example.com

go 1.20

require cuelang.org/go v0.6.0

-- deps.go --
// +build deps
package deps
import _ "cuelang.org/go/cmd/cue"
import _ "k8s.io/api"
import _ "k8s.io/apimachinery"

Use of MACHINERY=k8s.io/apimachinery/... fails in all scenarios:

With pre-generics K8S_VERSIONS because of cmd/cue: get go panics #668
With post-generics K8S_VERSIONS because of lack of generics support in cue get go (this issue), but even if we had that we would likely hit cmd/cue: get go panics #668

Use of MACHINERY=k8s.io/apimachinery/pkg/api/... succeeds with any recent GO_VERSION and CUE_VERSION, and with all the K8S_VERSION's that you listed above.

GO_VERSION is largely irrelevant, because generics have been supported as far back as 1.18. So I think that covers that angle.

CUE_VERSION is relevant insofar as when this issue is implemented, and #668 and others fixed, then things will behave correctly.

So the TL;DR from my perspective is:

We need to fix cmd/cue: get go panics #668 and others to avoid panics
We need to implemented this issue to support generics

Please let me know if there is anything that has fallen through the gaps in this analysis!

verdverm · 2023-08-17T06:04:16Z

Did you try cue get go k8s.io/apimachinery/...? I had to narrow (extend) the path /apimachinery/pkg/api/... to get it to work

myitcv · 2023-08-22T05:09:40Z

Did you try cue get go k8s.io/apimachinery/...?

Yes, per my comment above which gives the explanation as to why each fails:

Use of MACHINERY=k8s.io/apimachinery/... fails in all scenarios:

But what I then failed to do was actually comment that your solution of cue get go k8s.io/apimachinery/pkg/api/... does work. Thanks for pointing that out. I've updated my comment to indicate that.

rogpeppe added FeatureRequest New feature or request Triage Requires triage/attention labels Jan 16, 2023

mvdan removed the Triage Requires triage/attention label Jan 17, 2023

myitcv added the zGarden label Jun 13, 2023

mpvl added this to the Needs planning milestone Jun 14, 2023

myitcv removed the zGarden label Jun 14, 2023

mpvl added the get go issues related to cue get go label Jun 14, 2023

GeorgeMac mentioned this issue Jun 15, 2023

test(config): ensure default config passes CUE validation flipt-io/flipt#1758

Merged

cueckoo closed this as completed in 3b4e03d Oct 31, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cue/cmd: support generic Go types in `cue get go` #2217

cue/cmd: support generic Go types in `cue get go` #2217

rogpeppe commented Jan 16, 2023

mvdan commented Jan 17, 2023

myitcv commented Feb 2, 2023

uhthomas commented May 26, 2023

myitcv commented Jun 14, 2023

verdverm commented Aug 12, 2023 •

edited

Loading

verdverm commented Aug 12, 2023

verdverm commented Aug 13, 2023 •

edited

Loading

myitcv commented Aug 17, 2023 •

edited

Loading

verdverm commented Aug 17, 2023

myitcv commented Aug 22, 2023

cue/cmd: support generic Go types in cue get go #2217

cue/cmd: support generic Go types in cue get go #2217

Comments

rogpeppe commented Jan 16, 2023

mvdan commented Jan 17, 2023

myitcv commented Feb 2, 2023

uhthomas commented May 26, 2023

myitcv commented Jun 14, 2023

verdverm commented Aug 12, 2023 • edited Loading

verdverm commented Aug 12, 2023

verdverm commented Aug 13, 2023 • edited Loading

myitcv commented Aug 17, 2023 • edited Loading

verdverm commented Aug 17, 2023

myitcv commented Aug 22, 2023

cue/cmd: support generic Go types in `cue get go` #2217

cue/cmd: support generic Go types in `cue get go` #2217

verdverm commented Aug 12, 2023 •

edited

Loading

verdverm commented Aug 13, 2023 •

edited

Loading

myitcv commented Aug 17, 2023 •

edited

Loading