x/vgo: import package not yet uploaded to a remote location

[krasi-georgiev]

$GOPATH/src/github.com/prometheus/prombench
├── cmd
│   └── prombench
│       ├── go.mod
│       └── main.go
├── provider
│   └── gke
│       └── gke.go

runing vgo build inside ../cmd/prombench fails as
main.go imports github.com/prometheus/prombench/provider/gke
which is not yet uploaded to the remote repo since this is a WIP so can't yet upload it to upstream.

vgo: import "github.com/prometheus/prombench/cmd/prombench" ->
        import "github.com/prometheus/prombench/provider/gke" [/home/krasi/src/v/github.com/prometheus/prombench@v0.0.0-20180214143609-3b081cc497a5/provider/gke]: open /home/krasi/src/v/github.com/prometheus/prombench@v0.0.0-20180214143609-3b081cc497a5/provider/gke: no such file or directory

I tried adding replace "github.com/prometheus/prombench/provider/gke" v0.0.0 => "../../provider/gke" to go.mod , but didn't make any difference.

not sure what should be the behaviour here, but I see this as a quite common scenario

[myitcv]

Use a replace directive. I have exactly the same scenario here

[krasi-georgiev]

@myitcv I tried , but go build still fails with

vgo: import "github.com/prometheus/prombench/cmd/prombench" ->
        import "github.com/prometheus/prombench/provider/gke" [/home/krasi/src/v/github.com/prometheus/prombench@v0.0.0-20180214143609-3b081cc497a5/provider/gke]: open /home/krasi/src/v/github.com/prometheus/prombench@v0.0.0-20180214143609-3b081cc497a5/provider/gke: no such file or directory

[krasi-georgiev]

it looks like the first time vgo build ignores the replace directive.

[myitcv]

Do you have a go.mod at the repo root? If not I think you need one, e.g. mine is for myitcv.io. Yours would be for github.com/prometheus/prombench)

• If you intending for github.com/prometheus/prombench/provider/gke to be a module, then it will also need a go.mod (in provider/gke). In this case the replace directive from cmd/prombench would be "github.com/prometheus/prombench/provider/gke" => ../../provider/gke
• otherwise, the replace directive will be "github.com/prometheus/prombench" => ../../ (because provider/gke is a package within the main module

[myitcv]

If you are able to point us towards a branch of code that would probably help to diagnose.

[krasi-georgiev]

here is the WIP
https://github.com/krasi-georgiev/prombench/tree/vgo

tried few different replace options, but couldn't find the magic one.

[myitcv]

Ok I've just updated my earlier comment - my understanding is that you still need a go.mod in the root of your repo. And depending on whether github.com/prometheus/prombench/provider/gke will be a module or not it will also require a go.mod

So assuming you have:

$GOPATH/src/github.com/prometheus/prombench
└── go.mod
├── cmd
│   └── prombench
│       ├── go.mod
│       └── main.go
├── provider
│   └── gke
│       └── gke.go

I think there is actually a problem here because a vgo build in cmd/prombench ends up querying github.com when resolving github.com/prometheus/prombench/provider/gke; whereas I believe it should first walk to the repo root because that module (github.com/prometheus/prombench) is specified as a replace and indeed is a prefix of github.com/prometheus/prombench/cmd/prombench. I'll let @rsc comment on this though.

In the meantime, if you move your go.mod to the repo root, per https://github.com/myitcv/prombench/tree/vgo, then vgo build makes more progress (before failing somewhere in k8s land):

vgo: resolving import "github.com/pkg/errors"
vgo: finding github.com/pkg/errors (latest)
vgo: adding github.com/pkg/errors v0.8.0
vgo: resolving import "cloud.google.com/go/container/apiv1"
vgo: finding cloud.google.com/go (latest)
vgo: adding cloud.google.com/go v0.0.0-20180409203830-29f476ffa9c4
vgo: resolving import "google.golang.org/genproto/googleapis/container/v1"
vgo: finding google.golang.org/genproto (latest)
vgo: adding google.golang.org/genproto v0.0.0-20180409222037-51d0944304c3
vgo: resolving import "gopkg.in/alecthomas/kingpin.v2"
vgo: finding gopkg.in/alecthomas/kingpin.v2 (latest)
vgo: adding gopkg.in/alecthomas/kingpin.v2 v1.2.6-gopkgin-v2.2.6
vgo: resolving import "gopkg.in/yaml.v2"
vgo: finding gopkg.in/yaml.v2 (latest)
vgo: adding gopkg.in/yaml.v2 v1.2.1-gopkgin-v2.2.1
vgo: resolving import "k8s.io/api/apps/v1beta1"
...

[krasi-georgiev]

success!

for some reason vgo tries to use old version of some packages which caused build failures,but after adding the latest commit hashes manually it builds ok now.

if I comment out these 2 it tries to use older versions for some reason.

"golang.org/x/oauth2" v0.0.0-20180411000000-921ae394b9430ed4fb549668d7b087601bd60a81
"gopkg.in/yaml.v2" v0.0.0-20180411000000-5420a8b6744d3b0345ab293f6fcba19c978f1183

feel free to close the issue if you think it behaves as expected.

module "github.com/prometheus/prombench"

require (
	"cloud.google.com/go" v0.0.0-20180409203830-29f476ffa9c4
	"github.com/alecthomas/template" v0.0.0-20160405071501-a0175ee3bccc
	"github.com/alecthomas/units" v0.0.0-20151022065526-2efee857e7cf
	"github.com/googleapis/gax-go" v0.0.0-20180411000000-de2cc08e690b99dd3f7d19937d80d3d54e04682f
	"github.com/pkg/errors" v0.8.0
	"go.opencensus.io" v0.8.0
	"golang.org/x/oauth2" v0.0.0-20180411000000-921ae394b9430ed4fb549668d7b087601bd60a81
	"google.golang.org/api" v0.0.0-20180410000351-9f7560f3b05b
	"google.golang.org/genproto" v0.0.0-20180409222037-51d0944304c3
	"google.golang.org/grpc" v1.11.3
	"gopkg.in/alecthomas/kingpin.v2" v1.2.6-gopkgin-v2.2.6
	"gopkg.in/yaml.v2" v0.0.0-20180411000000-5420a8b6744d3b0345ab293f6fcba19c978f1183
	"k8s.io/api" v0.0.0-20180411000000-acf347b865f29325eb61f4cd2df11e86e073a5ee
	"k8s.io/apimachinery" v0.0.0-20180411000000-19e3f5aa3adca672c153d324e6b7d82ff8935f03
	"k8s.io/client-go" v0.0.0-20180411000000-78700dec6369ba22221b72770783300f143df150
	"k8s.io/kube-openapi" v0.0.0-20180410221256-abc8da71b985
)

[krasi-georgiev]

actually for yaml it is clear that the v2 doesn't have go.mod so probably that is why it defaults to v1 , but for oauth2 not sure why.

[myitcv]

I think there is actually a problem here because a vgo build in cmd/prombench ends up querying github.com when resolving github.com/prometheus/prombench/provider/gke; whereas I believe it should first walk to the repo root because that module (github.com/prometheus/prombench) is specified as a replace and indeed is a prefix of github.com/prometheus/prombench/cmd/prombench. I'll let @rsc comment on this though.

I've given this a bit more thought, and wonder whether what we need here, in addition to the replace directive, is the ability to specify the equivalent of a go-import meta tag override (which by definition would be at the package level):

metaoverride (
    // specify local directory
    "github.com/prometheus/prombench/provider/gke" => "github.com/prometheus/prombench/provider/gke" dir ../../

    // or specify git URL per regular go-import meta tag definition
    // the import path on the RHS required to specify the import
    // path at the root of the directory/URL
    "github.com/prometheus/prombench/provider/gke" => "github.com/prometheus/prombench" git https://github.com/myitcv/prombench
)

Resolutions would first check such overrides before hitting "the internet".

I'm fairly sure this idea isn't novel and that it will have been considered en route to the replace directive so would appreciate any pointers/explanations on where it falls down.

[krasi-georgiev]

isn't the problem here that vgo tries to download packages before it considers the replace directive?
what if vgo build checks the replace and if the file/folder exists than no need to try and download.