x/vgo: import package not yet uploaded to a remote location

[krasi-georgiev]

$GOPATH/src/github.com/prometheus/prombench
├── cmd
│   └── prombench
│       ├── go.mod
│       └── main.go
├── provider
│   └── gke
│       └── gke.go

runing vgo build inside ../cmd/prombench fails as
main.go imports github.com/prometheus/prombench/provider/gke
which is not yet uploaded to the remote repo since this is a WIP so can't yet upload it to upstream.

vgo: import "github.com/prometheus/prombench/cmd/prombench" ->
        import "github.com/prometheus/prombench/provider/gke" [/home/krasi/src/v/github.com/prometheus/prombench@v0.0.0-20180214143609-3b081cc497a5/provider/gke]: open /home/krasi/src/v/github.com/prometheus/prombench@v0.0.0-20180214143609-3b081cc497a5/provider/gke: no such file or directory

I tried adding replace "github.com/prometheus/prombench/provider/gke" v0.0.0 => "../../provider/gke" to go.mod , but didn't make any difference.

not sure what should be the behaviour here, but I see this as a quite common scenario

[myitcv]

Use a replace directive. I have exactly the same scenario here

[krasi-georgiev]

@myitcv I tried , but go build still fails with

vgo: import "github.com/prometheus/prombench/cmd/prombench" ->
        import "github.com/prometheus/prombench/provider/gke" [/home/krasi/src/v/github.com/prometheus/prombench@v0.0.0-20180214143609-3b081cc497a5/provider/gke]: open /home/krasi/src/v/github.com/prometheus/prombench@v0.0.0-20180214143609-3b081cc497a5/provider/gke: no such file or directory

[krasi-georgiev]

it looks like the first time vgo build ignores the replace directive.

[myitcv]

Do you have a go.mod at the repo root? If not I think you need one, e.g. mine is for myitcv.io. Yours would be for github.com/prometheus/prombench)

• If you intending for github.com/prometheus/prombench/provider/gke to be a module, then it will also need a go.mod (in provider/gke). In this case the replace directive from cmd/prombench would be "github.com/prometheus/prombench/provider/gke" => ../../provider/gke
• otherwise, the replace directive will be "github.com/prometheus/prombench" => ../../ (because provider/gke is a package within the main module

[myitcv]

If you are able to point us towards a branch of code that would probably help to diagnose.

[krasi-georgiev]

here is the WIP
https://github.com/krasi-georgiev/prombench/tree/vgo

tried few different replace options, but couldn't find the magic one.

[myitcv]

Ok I've just updated my earlier comment - my understanding is that you still need a go.mod in the root of your repo. And depending on whether github.com/prometheus/prombench/provider/gke will be a module or not it will also require a go.mod

So assuming you have:

$GOPATH/src/github.com/prometheus/prombench
└── go.mod
├── cmd
│   └── prombench
│       ├── go.mod
│       └── main.go
├── provider
│   └── gke
│       └── gke.go

I think there is actually a problem here because a vgo build in cmd/prombench ends up querying github.com when resolving github.com/prometheus/prombench/provider/gke; whereas I believe it should first walk to the repo root because that module (github.com/prometheus/prombench) is specified as a replace and indeed is a prefix of github.com/prometheus/prombench/cmd/prombench. I'll let @rsc comment on this though.

In the meantime, if you move your go.mod to the repo root, per https://github.com/myitcv/prombench/tree/vgo, then vgo build makes more progress (before failing somewhere in k8s land):

vgo: resolving import "github.com/pkg/errors"
vgo: finding github.com/pkg/errors (latest)
vgo: adding github.com/pkg/errors v0.8.0
vgo: resolving import "cloud.google.com/go/container/apiv1"
vgo: finding cloud.google.com/go (latest)
vgo: adding cloud.google.com/go v0.0.0-20180409203830-29f476ffa9c4
vgo: resolving import "google.golang.org/genproto/googleapis/container/v1"
vgo: finding google.golang.org/genproto (latest)
vgo: adding google.golang.org/genproto v0.0.0-20180409222037-51d0944304c3
vgo: resolving import "gopkg.in/alecthomas/kingpin.v2"
vgo: finding gopkg.in/alecthomas/kingpin.v2 (latest)
vgo: adding gopkg.in/alecthomas/kingpin.v2 v1.2.6-gopkgin-v2.2.6
vgo: resolving import "gopkg.in/yaml.v2"
vgo: finding gopkg.in/yaml.v2 (latest)
vgo: adding gopkg.in/yaml.v2 v1.2.1-gopkgin-v2.2.1
vgo: resolving import "k8s.io/api/apps/v1beta1"
...

[krasi-georgiev]

success!

for some reason vgo tries to use old version of some packages which caused build failures,but after adding the latest commit hashes manually it builds ok now.

if I comment out these 2 it tries to use older versions for some reason.

"golang.org/x/oauth2" v0.0.0-20180411000000-921ae394b9430ed4fb549668d7b087601bd60a81
"gopkg.in/yaml.v2" v0.0.0-20180411000000-5420a8b6744d3b0345ab293f6fcba19c978f1183

feel free to close the issue if you think it behaves as expected.

module "github.com/prometheus/prombench"

require (
	"cloud.google.com/go" v0.0.0-20180409203830-29f476ffa9c4
	"github.com/alecthomas/template" v0.0.0-20160405071501-a0175ee3bccc
	"github.com/alecthomas/units" v0.0.0-20151022065526-2efee857e7cf
	"github.com/googleapis/gax-go" v0.0.0-20180411000000-de2cc08e690b99dd3f7d19937d80d3d54e04682f
	"github.com/pkg/errors" v0.8.0
	"go.opencensus.io" v0.8.0
	"golang.org/x/oauth2" v0.0.0-20180411000000-921ae394b9430ed4fb549668d7b087601bd60a81
	"google.golang.org/api" v0.0.0-20180410000351-9f7560f3b05b
	"google.golang.org/genproto" v0.0.0-20180409222037-51d0944304c3
	"google.golang.org/grpc" v1.11.3
	"gopkg.in/alecthomas/kingpin.v2" v1.2.6-gopkgin-v2.2.6
	"gopkg.in/yaml.v2" v0.0.0-20180411000000-5420a8b6744d3b0345ab293f6fcba19c978f1183
	"k8s.io/api" v0.0.0-20180411000000-acf347b865f29325eb61f4cd2df11e86e073a5ee
	"k8s.io/apimachinery" v0.0.0-20180411000000-19e3f5aa3adca672c153d324e6b7d82ff8935f03
	"k8s.io/client-go" v0.0.0-20180411000000-78700dec6369ba22221b72770783300f143df150
	"k8s.io/kube-openapi" v0.0.0-20180410221256-abc8da71b985
)

[krasi-georgiev]

actually for yaml it is clear that the v2 doesn't have go.mod so probably that is why it defaults to v1 , but for oauth2 not sure why.

[myitcv]

I think there is actually a problem here because a vgo build in cmd/prombench ends up querying github.com when resolving github.com/prometheus/prombench/provider/gke; whereas I believe it should first walk to the repo root because that module (github.com/prometheus/prombench) is specified as a replace and indeed is a prefix of github.com/prometheus/prombench/cmd/prombench. I'll let @rsc comment on this though.

I've given this a bit more thought, and wonder whether what we need here, in addition to the replace directive, is the ability to specify the equivalent of a go-import meta tag override (which by definition would be at the package level):

metaoverride (
    // specify local directory
    "github.com/prometheus/prombench/provider/gke" => "github.com/prometheus/prombench/provider/gke" dir ../../

    // or specify git URL per regular go-import meta tag definition
    // the import path on the RHS required to specify the import
    // path at the root of the directory/URL
    "github.com/prometheus/prombench/provider/gke" => "github.com/prometheus/prombench" git https://github.com/myitcv/prombench
)

Resolutions would first check such overrides before hitting "the internet".

I'm fairly sure this idea isn't novel and that it will have been considered en route to the replace directive so would appreciate any pointers/explanations on where it falls down.

[krasi-georgiev]

isn't the problem here that vgo tries to download packages before it considers the replace directive?
what if vgo build checks the replace and if the file/folder exists than no need to try and download.

[myitcv]

isn't the problem here that vgo tries to download packages before it concideres the replace directive?

I don't think so; because there's a necessary resolution step (assuming there's nothing in the cache) where vgo needs to understand where pkg X lives. The return go-import meta directive gives that answer; and if that answer is module Y which happens to then have a replace directive, that's when it (the replace directive) kicks in.

Hence why the proposal for the go-import meta tag override.

cc @rogpeppe

[krasi-georgiev]

from my perspective without knowing the technical implementations it seems that metaoverride and replace overlap its meanings.
shouldn't replace always affect what package is used?

[myitcv]

Many packages can live within a module. And indeed modules can live within the repo of other modules (i.e. they are submodules).

My understanding is that replace operates on modules. In working out what module a package lives inside (and the package itself could be the module), vgo goes through a resolution step. Hence why I don't think the overlap you describe exists.

Actually, the concept of a go-import meta tag override only applies in the case of custom domains (which is my situation, hence the bias). But this idea of a go-import-esque package-level override could apply regardless of whether we're talking a custom domain, code host etc.

[krasi-georgiev]

I would need to spend a bit more time using vgo and see few more examples to fully understand this idea.

[rsc]

I'm sorry for coming to this conversation late, but @krasi-georgiev can you say a little more about why you want to structure prometheus/prombench as multiple modules?

[paralin]

I need the same thing, in this case, because of a domain that is down. There is an import path in one of our dependencies for "leb.io/hashland/keccakpg" and "leb.io" has been down since May. I'd love to remap it to the GitHub path, but vgo tries to download before evaluating the replacement:

vgo: resolving import "leb.io/hashland/keccakpg"
FindRepo: Get https://leb.io/hashland/keccakpg?go-get=1: dial tcp 104.131.190.18:443: connect: connection 
refused                                                                                                  
FindRepo: Get https://leb.io/hashland?go-get=1: dial tcp 104.131.190.18:443: connect: connection refused
FindRepo: Get https://leb.io?go-get=1: dial tcp 104.131.190.18:443: connect: connection refused
vgo: import "github.com/faceit/matcher-go/p2p" ->
        import "github.com/libp2p/go-libp2p" ->
        import "github.com/libp2p/go-libp2p-circuit" ->
        import "github.com/libp2p/go-libp2p-host" ->
        import "github.com/libp2p/go-libp2p-interface-connmgr" ->
        import "github.com/libp2p/go-libp2p-net" ->
        import "github.com/libp2p/go-libp2p-peer" ->
        import "github.com/multiformats/go-multihash" ->
        import "leb.io/hashland/keccakpg": Get https://leb.io/hashland/keccakpg?go-get=1: dial tcp 104.131
.190.18:443: connect: connection refused                                                                 

[krasi-georgiev]

@rsc actually I don't want multiple modules , I just thought that the structure of the project should be as per my first comment #24804 (comment) . So After moving go.mod in the project root it all works except a small nit as in my comment #24804 (comment)

[brunetto]

Hi,
I'm still testing vgo and trying to understand how I should work with it, so please forgive me if I'm missing something.

My use case, related to this issues, is:

• write down some code to explore an idea
• not able to set-up a repository online now (e.g. offline)
• don't want to set-up a repository online now (e.g. I don't want to create and online repo every time I explore an idea)

I am not able to run vgo build successfully until all the following happen

• I create the repo online
• I'm able to connect to it
• I upload the code at least once (otherwise I obtain no Go source files)

even if I run vgo build on the module (github.com/brunetto/vigo).
Nothing works until vgo is able to "see" the source code online. No way to run it fully local.

A reference repo is here: https://github.com/brunetto/vigo.

[rsc]

@brunetto, your repo is very close. The only missing bit is in cmd/vigo/go.mod there is no "require" statement asking for the root repo. The "replace" only kicks in if the requirement exists (because in general replace could be replacing an indirect requirement, not just things directly written in the current effective go.mod).

If I do this in go.mod then it works fine:

module github.com/brunetto/vigo/cmd/vigo

replace github.com/brunetto/vigo v0.0.99 => ../../../vigo

require github.com/brunetto/vigo v0.0.99

I just made up v0.0.99 - obviously that doesn't exist online right now.

Every command does print go: finding github.com/brunetto/vigo v0.0.99 but I'll get rid of that print (and the network lookup). It was already on my list.