Glide seems to hang with high CPU importing kubernetes API

[readams]

Create:

$GOPATH/src/example.com/example/main.go:

package main

import (
    "k8s.io/kubernetes/pkg/client/restclient"
    client "k8s.io/kubernetes/pkg/client/unversioned"
)

func main() {
    var config *restclient.Config
    client.New(config)
}

Now run:

cd $GOPATH/src/example.com/example
glide create --non-interactive
glide install

It will make slow progress for a while but eventually seems to hits a wall and seems to make no progress. It's using about 1 full CPU core while resident size increases only very slowly.

I'm using go 1.6 on Ubuntu Xenial and glide 0.12.1 from ppa.

Output looks like:

$ glide init --non-interactive
[INFO]  Generating a YAML configuration file and guessing the dependencies
[INFO]  Attempting to import from other package managers (use --skip-import to skip)
[INFO]  Scanning code to look for dependencies
[INFO]  --> Found reference to k8s.io/kubernetes/pkg/client/restclient
[INFO]  --> Adding sub-package pkg/client/unversioned to k8s.io/kubernetes
[INFO]  Writing configuration file (glide.yaml)
[INFO]  You can now edit the glide.yaml file. Consider:
[INFO]  --> Using versions and ranges. See https://glide.sh/docs/versions/
[INFO]  --> Adding additional metadata. See https://glide.sh/docs/glide.yaml/
[INFO]  --> Running the config-wizard command to improve the versions in your configuration

$ glide install
[INFO]  Lock file (glide.lock) does not exist. Performing update.
[INFO]  Downloading dependencies. Please wait...
[INFO]  --> Fetching updates for k8s.io/kubernetes.
[INFO]  Resolving imports
[INFO]  --> Fetching updates for github.com/golang/glog.
[INFO]  --> Fetching updates for github.com/google/cadvisor.
[INFO]  --> Fetching updates for github.com/davecgh/go-spew.
[INFO]  --> Fetching updates for github.com/ugorji/go.
[INFO]  --> Fetching updates for golang.org/x/net.
[INFO]  --> Fetching updates for github.com/go-openapi/spec.
[INFO]  --> Fetching updates for github.com/gogo/protobuf.
[INFO]  --> Fetching updates for github.com/google/gofuzz.
[INFO]  --> Fetching updates for github.com/juju/ratelimit.
[INFO]  --> Fetching updates for github.com/blang/semver.
[INFO]  --> Fetching updates for github.com/emicklei/go-restful.
[INFO]  --> Fetching updates for github.com/spf13/pflag.
[INFO]  --> Fetching updates for gopkg.in/inf.v0.
[INFO]  --> Fetching updates for github.com/pborman/uuid.
[INFO]  --> Fetching updates for github.com/go-openapi/jsonpointer.
[INFO]  --> Fetching updates for github.com/go-openapi/jsonreference.
[INFO]  --> Fetching updates for github.com/go-openapi/swag.
[INFO]  --> Fetching updates for github.com/docker/distribution.
[INFO]  --> Fetching updates for golang.org/x/oauth2.
[INFO]  --> Fetching updates for k8s.io/client-go.
[INFO]  --> Fetching updates for github.com/coreos/go-oidc.
[INFO]  --> Fetching updates for github.com/ghodss/yaml.
[INFO]  --> Fetching updates for github.com/PuerkitoBio/purell.
[INFO]  --> Fetching updates for github.com/mailru/easyjson.
[INFO]  --> Fetching updates for google.golang.org/appengine.
[INFO]  --> Fetching updates for cloud.google.com/go.
[INFO]  --> Fetching updates for github.com/coreos/pkg.
[INFO]  --> Fetching updates for github.com/jonboulle/clockwork.
[INFO]  --> Fetching updates for github.com/cloudfoundry-incubator/candiedyaml.
[INFO]  --> Fetching updates for github.com/PuerkitoBio/urlesc.
[INFO]  --> Fetching updates for golang.org/x/text.

[seh]

I think I've run into the same problem using Glide version version 0.12.2. Running glide install, after Glide fetches various repositories, it grinds away emitting messages like the following:

[DEBUG] Package github.com/go-openapi/spec imports github.com/go-openapi/jsonpointer
[DEBUG] Missing github.com/go-openapi/jsonpointer. Trying to resolve.
[DEBUG] Dependency github.com/go-openapi/jsonpointer has already been pinned. Fetching updates skipped.
[DEBUG] Dependency github.com/go-openapi/jsonpointer has already been pinned. Setting version skipped.
[DEBUG] Package github.com/go-openapi/spec imports github.com/go-openapi/jsonreference
[DEBUG] Missing github.com/go-openapi/jsonreference. Trying to resolve.
[DEBUG] Dependency github.com/go-openapi/jsonreference has already been pinned. Fetching updates skipped.
[DEBUG] Dependency github.com/go-openapi/jsonreference has already been pinned. Setting version skipped.
[DEBUG] Package github.com/go-openapi/spec imports github.com/go-openapi/swag
[DEBUG] Missing github.com/go-openapi/swag. Trying to resolve.
[DEBUG] Dependency github.com/go-openapi/swag has already been pinned. Fetching updates skipped.
[DEBUG] Dependency github.com/go-openapi/swag has already been pinned. Setting version skipped.
[DEBUG] Trying to open github.com/gogo/protobuf/proto (/Users/seh/.glide/cache/src/https-github.com-gogo-protobuf/proto)
[DEBUG] Trying to open github.com/gogo/protobuf/sortkeys (/Users/seh/.glide/cache/src/https-github.com-gogo-protobuf/sortkeys)
[DEBUG] Trying to open github.com/google/gofuzz (/Users/seh/.glide/cache/src/https-github.com-google-gofuzz)
[DEBUG] Trying to open k8s.io/kubernetes/pkg/conversion (/Users/seh/.glide/cache/src/https-github.com-kubernetes-kubernetes/pkg/conversion)
[DEBUG] Package k8s.io/kubernetes/pkg/conversion imports k8s.io/kubernetes/third_party/forked/golang/reflect
[DEBUG] Missing k8s.io/kubernetes/third_party/forked/golang/reflect. Trying to resolve.
[DEBUG] Dependency k8s.io/kubernetes has already been pinned. Fetching updates skipped.
[DEBUG] Dependency k8s.io/kubernetes has already been pinned. Setting version skipped.
[DEBUG] Trying to open k8s.io/kubernetes/pkg/genericapiserver/openapi/common (/Users/seh/.glide/cache/src/https-github.com-kubernetes-kubernetes/pkg/genericapiserver/openapi/common)
[DEBUG] Package k8s.io/kubernetes/pkg/genericapiserver/openapi/common imports github.com/go-openapi/spec
[DEBUG] Missing github.com/go-openapi/spec. Trying to resolve.
[DEBUG] Dependency github.com/go-openapi/spec has already been pinned. Fetching updates skipped.
[DEBUG] Dependency github.com/go-openapi/spec has already been pinned. Setting version skipped.
[DEBUG] Trying to open k8s.io/kubernetes/pkg/labels (/Users/seh/.glide/cache/src/https-github.com-kubernetes-kubernetes/pkg/labels)

I've had it running continuously now for about 15 hours, and I can't tell if it's getting any closer to finishing.

[timoreimann]

FWIW, I don't think I have experienced this problem with Glide 0.11 and my Kubernetes dependencies have been about the same for quite a while.

[seh]

Per @mattfarina's comment in the "general" channel of the "Gophers" Slack team, there's a symbolic link in the Kubernetes repository that's aggravating Glide's ability to finish:

% ls -l ~/.glide/cache/src/https-github.com-kubernetes-kubernetes/vendor/k8s.io 
total 8
lrwxr-xr-x  1 seh  staff   34 Sep 23 13:26 client-go -> ../../staging/src/k8s.io/client-go
drwxr-xr-x  4 seh  staff  136 Sep 23 13:26 heapster

[seh]

I find that the same problem arises when my project depends on just the kubernetes/client-go repository, so I don't think that this symbolic link is the problem here.

[mattfarina]

Can you try the fix to Glide in PR #622? I think this fixes it.

[seh]

I'll clone that and try it, but I may not finish before I have to give up tonight. I'll let you know by tomorrow morning.

[timoreimann]

@mattfarina @seh I just checked out PR #622 and used it to repopulate my sample Kubernetes client's dependencies (among which k8s.io/client-go is).

glide update ran to completion just fine again (whereas v0.11 would consistently loop forever). 🎉

From my perspective, this is ready to 🚢 !