cmd/go: unexpected build failure with modules

[hoyho]

What version of Go are you using (go version)?

$ go version
go version go1.11 darwin/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/xsky/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/xsky/Documents/workspace/gopath"
GOPROXY=""
GORACE=""
GOROOT="/Users/xsky/Documents/program/go"
GOTMPDIR=""
GOTOOLDIR="/Users/xsky/Documents/program/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/xsky/Documents/tmp/xsky-csi-mod/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/4_/fcwszxr93jzc96dp3pzttx3h0000gn/T/go-build753346766=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

copy a project to a folder outside of my GOPATH

go mod init github.example.com/xsky/xsky-csi
go clean -modcache
go build main.go

What did you expect to see?

build successful

What did you see instead?

build fail with output：

# k8s.io/apiextensions-apiserver/pkg/features
../../workspace/gopath/pkg/mod/k8s.io/apiextensions-apiserver@v0.0.0-20190119024419-80a4532647cb/pkg/features/kube_features.go:52:2: undefined: feature.DefaultMutableFeatureGate

content in go.mod:

module github.xsky.com/xsky/xsky-csi

require (
	github.com/NYTimes/gziphandler v1.0.1 // indirect
	github.com/container-storage-interface/spec v1.0.0
	github.com/coreos/etcd v3.3.11+incompatible // indirect
	github.com/coreos/go-systemd v0.0.0-20181031085051-9002847aa142 // indirect
	github.com/docker/distribution v2.7.1+incompatible // indirect
	github.com/docker/docker v1.13.1 // indirect
	github.com/elazarl/go-bindata-assetfs v1.0.0 // indirect
	github.com/emicklei/go-restful v2.8.1+incompatible // indirect
	github.com/emicklei/go-restful-swagger12 v0.0.0-20170926063155-7524189396c6 // indirect
	github.com/evanphx/json-patch v4.1.0+incompatible // indirect
	github.com/go-openapi/spec v0.18.0 // indirect
	github.com/go-openapi/strfmt v0.18.0 // indirect
	github.com/go-openapi/validate v0.18.0 // indirect
	github.com/gogo/protobuf v1.2.0 // indirect
	github.com/golang/groupcache v0.0.0-20181024230925-c65c006176ff // indirect
	github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c // indirect
	github.com/google/gofuzz v0.0.0-20170612174753-24818f796faf // indirect
	github.com/googleapis/gnostic v0.2.0 // indirect
	github.com/gregjones/httpcache v0.0.0-20181110185634-c63ab54fda8f // indirect
	github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 // indirect
	github.com/hashicorp/golang-lru v0.5.0 // indirect
	github.com/imdario/mergo v0.3.6 // indirect
	github.com/json-iterator/go v1.1.5 // indirect
	github.com/kubernetes-csi/drivers v0.5.0-alpha.0.0.20181129192734-8a7f2d3a4057 //v1.0.2
	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
	github.com/modern-go/reflect2 v1.0.1 // indirect
	github.com/munnerz/goautoneg v0.0.0-20120707110453-a547fc61f48d // indirect
	github.com/onsi/ginkgo v1.7.0 // indirect
	github.com/onsi/gomega v1.4.3 // indirect
	github.com/opencontainers/go-digest v1.0.0-rc1 // indirect
	github.com/pborman/uuid v1.2.0
	github.com/peterbourgon/diskv v2.0.1+incompatible // indirect
	github.com/prometheus/client_golang v0.9.2 // indirect
	github.com/sirupsen/logrus v1.3.0
	github.com/spf13/afero v1.2.0 // indirect
	github.com/spf13/cobra v0.0.3 // indirect
	github.com/spf13/pflag v1.0.3 // indirect
	golang.org/x/net v0.0.0-20190119204137-ed066c81e75e
	golang.org/x/oauth2 v0.0.0-20190115181402-5dab4167f31c
	golang.org/x/time v0.0.0-20181108054448-85acf8d2951c // indirect
	google.golang.org/grpc v1.18.0
	gopkg.in/inf.v0 v0.9.1 // indirect
	gopkg.in/natefinch/lumberjack.v2 v2.0.0 // indirect
	gopkg.in/yaml.v2 v2.2.2 // indirect
	k8s.io/api v0.0.0-20181221193117-173ce66c1e39 // indirect
	k8s.io/apiextensions-apiserver v0.0.0-20190119024419-80a4532647cb
	k8s.io/apimachinery v0.0.0-20190119020841-d41becfba9ee // indirect
	k8s.io/apiserver v0.0.0-20190111033246-d50e9ac5404f+incompatible // indirect;
	k8s.io/client-go v10.0.0+incompatible // indirect
	k8s.io/cloud-provider v0.0.0-20181221204816-2325825fd8d8 // indirect
	k8s.io/component-base v0.0.0-20190118114819-7b44618cd005 // indirect
	k8s.io/csi-api v0.0.0-20190119024756-9225cacc8063 // indirect
	k8s.io/klog v0.1.0 // indirect
	k8s.io/kube-openapi v0.0.0-20190115222348-ced9eb3070a5 // indirect
	k8s.io/kubernetes v1.13.0 // v1.13.2
	k8s.io/utils v0.0.0-20181221173059-8a16e7dd8fb6
	sigs.k8s.io/yaml v1.1.0 // indirect
)

try to click the link from console error

I try to trace the dependence of apiserver@v0.0.0-20190111033246-d50e9ac5404f+incompatible/pkg/util/feature/feature_gate.go and found there's function Add()..

is the dependence(apiextensions-apiserver@v0.0.0-20190119024419-80a4532647cb) can not find its correct dependence(k8s.io/apiserver/pkg/util/feature)?
what 's the main reason it can not build, and how to solve it?
thanks

[hoyho]

maybe the problem is that get the incorrect version of Kubernetes@1.13.2

code from gopath/pkg/mod/k8s.io/kubernetes@v1.13.2

i check the latest version of kubernetes at github:

is it? is my go mod get ther wrong kubernetes@v1.13.2?

[FiloSottile]

/cc @bcmills

[bcmills]

The problem here is some sort of version skew between the various kubernetes modules. (See https://github.com/kubernetes/apiextensions-apiserver#compatibility.)

I'm not sure exactly which module is wrong, but it doesn't really matter: you can resolve it by running

go get -m k8s.io/apiserver@kubernetes-1.14.0-alpha.1 k8s.io/apiextensions-apiserver@kubernetes-1.14.0-alpha.1 k8s.io/apimachinery@kubernetes-1.14.0-alpha.1 k8s3io/client-go@kubernetes-1.14.0-alpha.1

xksy-csi$ cat >./main.go <<EOF
package main
import _ "k8s.io/apiextensions-apiserver/pkg/features"
func main() {}
EOF

xksy-csi$ cat >./go.mod <<EOF
module github.xsky.com/xsky/xsky-csi

require (
        github.com/davecgh/go-spew v1.1.1 // indirect
        github.com/pmezard/go-difflib v1.0.0 // indirect
        github.com/spf13/pflag v1.0.3 // indirect
        github.com/stretchr/testify v1.2.2 // indirect
        k8s.io/apiextensions-apiserver v0.0.0-20190119024419-80a4532647cb
        k8s.io/apiserver v0.0.0-20190111033246-d50e9ac5404f+incompatible // indirect
        k8s.io/klog v0.1.0 // indirect
)
EOF

xksy-csi$ cat >./go.sum <<EOF
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/spf13/pflag v1.0.3 h1:zPAT6CGy6wXeQ7NtTnaTerfKOsV6V6F8agHXFiazDkg=
github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
github.com/stretchr/testify v1.2.2 h1:bSDNvY7ZPG5RlJ8otE/7V6gMiyenm9RtJ7IUVIAoJ1w=
github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
k8s.io/apiextensions-apiserver v0.0.0-20190115195204-f00e89d6e347 h1:eu7UbWF4fXkf34ue43rGiAu+S1ynDchohsVVo5pZtp8=
k8s.io/apiextensions-apiserver v0.0.0-20190115195204-f00e89d6e347/go.mod h1:IxkesAMoaCRoLrPJdZNZUQp9NfZnzqaVzLhb2VEQzXE=
k8s.io/apiserver v0.0.0-20190115192846-d7664e065129 h1:BhZHOXAP5aJoWt63Go/Rgn0h9sJ0FTTAcWylu4EaY3Y=
k8s.io/apiserver v0.0.0-20190115192846-d7664e065129/go.mod h1:6bqaTSOSJavUIXUtfaR9Os9JtTCm8ZqH2SUl2S60C4w=
k8s.io/klog v0.1.0 h1:I5HMfc/DtuVaGR1KPwUrTc476K8NCqNBldC7H4dYEzk=
k8s.io/klog v0.1.0/go.mod h1:Gq+BEi5rUBO/HRz0bTSXDUcqjScdoY3a9IHpCEIOOfk=
EOF

xksy-csi$ cat >./go.mod.orig <<EOF
module github.xsky.com/xsky/xsky-csi

require (
        github.com/davecgh/go-spew v1.1.1 // indirect
        github.com/pmezard/go-difflib v1.0.0 // indirect
        github.com/spf13/pflag v1.0.3 // indirect
        github.com/stretchr/testify v1.2.2 // indirect
        k8s.io/apiextensions-apiserver v0.0.0-20190119024419-80a4532647cb
        k8s.io/apiserver v0.0.0-20190111033246-d50e9ac5404f+incompatible // indirect
        k8s.io/klog v0.1.0 // indirect
)
EOF

xksy-csi$ cp go.mod go.mod.orig

xksy-csi$ go build k8s.io/apiextensions-apiserver/pkg/features
# k8s.io/apiextensions-apiserver/pkg/features
../_gopath/pkg/mod/k8s.io/apiextensions-apiserver@v0.0.0-20190119024419-80a4532647cb/pkg/features/kube_features.go:52:2: undefined: feature.DefaultMutableFeatureGate

xksy-csi$ go get -m k8s.io/apiserver@kubernetes-1.14.0-alpha.1 k8s.io/apiextensions-apiserver@kubernetes-1.14.0-alpha.1 k8s.io/apimachinery@kubernetes-1.14.0-alpha.1
go: finding k8s.io/apimachinery kubernetes-1.14.0-alpha.1
go: finding k8s.io/apiserver kubernetes-1.14.0-alpha.1
go: finding k8s.io/apiextensions-apiserver kubernetes-1.14.0-alpha.1

xksy-csi$ go mod tidy

xksy-csi$ go build k8s.io/apiextensions-apiserver/pkg/features

xksy-csi$ diff go.mod.orig go.mod
8,9c8,9
<       k8s.io/apiextensions-apiserver v0.0.0-20190119024419-80a4532647cb
<       k8s.io/apiserver v0.0.0-20190111033246-d50e9ac5404f+incompatible // indirect
---
>       k8s.io/apiextensions-apiserver v0.0.0-20190115195204-f00e89d6e347
>       k8s.io/apiserver v0.0.0-20190115192846-d7664e065129 // indirect

xksy-csi$

[hoyho]

thanks @bcmills , you save my day, in my case, also need to upgrade kubernetes
go get -v -m k8s.io/kubernetes@v1.14.0-alpha.1
and add it to go.mod's require
then it build successful

however, i still can't get kubernetes@v1.13.0 or 1.13.2 or else
what if i have to use kubernetes@v1.13.0? can I ?

[bcmills]

however, i still can't get kubernetes@v1.13.0 or 1.13.2 or else
what if i have to use kubernetes@v1.13.0? can I ?

I have no idea. Presumably there is some 1.13 version at which all of the kubernetes packages are compatible, but are you sure that none of your other dependencies require features introduced in 1.14?

[thepudds]

@hoyho

k8s has a relatively complex and non-standard way to manage dependencies including custom wrapper scripts on top of godep, I think.

If you want to get a consistent set of v1.13.2 dependecies for k8s, one approach you could try is cloning
something like github.com/kubernetes/kubernetes, and then try to use the dependency information from Godeps.json:
https://github.com/kubernetes/kubernetes/blob/v1.13.2/Godeps/Godeps.json

You might be able to have go mod init do most of the work for you. For example, could be something along the lines of:

$ git clone https://github.com/kubernetes/kubernetes
$ cd kubernetes/
$ go mod init temp
go: creating new go.mod: module temp
go: copying requirements from Godeps\Godeps.json

$ cat go.mod | wc -l
214

The following FAQ covers the technique in a bit more detail:

"FAQ: I have a problem with a complex dependency that has not opted in to modules. Can I use information from its current dependency manager?"

Here is a concrete example in #28489 (comment) of following that technique (for docker, in that example).