cmd/go: custom import path private repos require special configuration

[chowey]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.11 darwin/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOOS="darwin"

What did you do?

I have a private repo served over ssh, using gitolite.

That private repo is a package I want to use, at e.g. private.repo.net/user/package. It has a folder structure like this:

README.md
a/a.go
b/b.go

My global .gitconfig looks like this, so that go get works:

[url "ssh://git@private.repo.net"]
        insteadOf = https://private.repo.net

In my main package my/package, I have a go file that imports it like this:

import "private.repo.net/user/package/a"

This used to work when using the old $GOPATH. Now I am trying to use modules instead. I run go build my/package which should resolve dependencies and build.

What did you expect to see?

I expected it to build.

What did you see instead?

build my/package: cannot find module for path private.repo.net/user/package/a

I also tried adding it to the go.mod file by running go get private.repo.net/user/package.git. (NOTE: I need to use .git so that it uses ssh.) So it now private.repo.net/user/package.git appears in my go.mod file.

However, when I do go build, it still complains that "build my/package: cannot find module for path private.repo.net/user/package/a".

[chowey]

It works if I just do:

import "private.repo.net/user/package.git/a"

(emphasis on ".git") Then go build does the right thing, and correctly initializes the go.mod file.

NOTE: I still need to do git config --global url."ssh://git@private.repo.net".insteadOf "https://private.repo.net" to get this to work.

[lopezator]

Same error here, import paths like:

my-private-repo.com/user/pkg/a

Simply don't work and prints:

cannot find module for path ....

[myitcv]

@chowey the reason the .git solution works is that it helps guide the go tool with information about where the VCS for custom import paths private.repo.net/user/package.git/... resides (namely (https|ssh)://private.repo.net/user/package.git). Note for github.com and other VCS's known to the go tool, this step is not required.

Without this (i.e. with an import path like private.repo.net/user/package/a), the go tool first queries https://private.repo.net/user/package/a?go-get=1 to work out (according to https://tip.golang.org/cmd/go/#hdr-Remote_import_paths) where the VCS for that custom import path lives. I suspect this is where yours and @lopezator's examples are failing.

Hence you need to ensure:

• https://private.repo.net/user/package/a?go-get=1 returns <meta> information per https://tip.golang.org/cmd/go/#hdr-Remote_import_paths
• https://private.repo.net/user/package/a?go-get=1 must also be accessible; either this means that it returns the <meta> information regardless of auth credentials, or you need some sort of credentials helper

For credentials, either you can provide a $HOME/.netrc:

machine private.repo.net login YOU password APIKEY

or use an OS credential helper (see #26134 (comment)).

Also related to #26232.

Whether you then use https/ssh access to the underlying VCS is entirely up to you.

---

I think this issue is effectively the custom import path equivalent of #26134. Hence I'm going to retitle it.

[jackwhelpton]

I'm not sure if this is germane to this issue or I should open a new one, but I seem to be able to reproduce this despite checking all of the configuration above:

• even non-nested imports (import "private.repo.net/user/package") are failing with cannot find module for path

• this behavior is only apparent with GO111MODULE=on; it does not manifest when GO111MODULE=off

• I have this in my .gitconfig:

  [url "git@private.repo.net"]
  insteadOf = https://private.repo.net

• I'm running on Windows, and have the credential manager configured. I can pull from the repo fine.

• I've checked the meta tags from https://private.repo.net/user/package?go-get=1 and they look fine.

[myitcv]

@jackwhelpton are you able to share the output of go get -x SOMETHING?

This might well contain sensitive information in your case, so please be careful.

In case you can't share this information publicly, please do say.

[jackwhelpton]

Sure. It does look like an authentication problem, but my standard git commands are working OK. Here's what I get from go get -x private.repo.net/user/package

go get -x private.repo.net/user/package
# C:\Projects\go\pkg\mod\cache\vcs\<id> for git2 https://private.repo.net/user/package.git
cd C:\Projects\go\pkg\mod\cache\vcs\<id>; git ls-remote -q https://private.repo.net/user/package.git
0.742s # cd C:\Projects\go\pkg\mod\cache\vcs\<id>; git ls-remote -q https://private.repo.net/user/package.git
go get private.repo.net/user/package: git ls-remote -q https://private.repo.net/user/package.git in C:\Projects\go\pkg\mod\cache\vcs\<id>: exit status 128:
        fatal: 'git@private.repo.net/user/package.git' does not appear to be a git repository
        fatal: Could not read from remote repository.

        Please make sure you have the correct access rights
        and the repository exists.

[myitcv]

@jackwhelpton if you run

git ls-remote -q https://private.repo.net/user/package.git

separately, what is the output?

[jackwhelpton]

Ah, so this may be a git configuration issue, although it's interacting with modules in a strange way (given my ability to go get successfully when modules are disabled). Here's what I see with that command:

git ls-remote -q https://private.repo.net/user/package.git
fatal: 'git@private.repo.net/user/package.git' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Note that a simple git pull in that directory still succeeds. I've just realized there's one piece of configuration I hadn't explained yet: I'm interacting with that server using a stored SSH key, and PuTTY/Pageant is handling the key exchange.

[myitcv]

@jackwhelpton your insteadOf configuration does not look correct:

Try:

[url "ssh://git@private.repo.net"]
insteadOf = https://private.repo.net

instead of (no pun intended):

[url "git@private.repo.net"]
insteadOf = https://private.repo.net

[jackwhelpton]

That does it! Good spot. Should have guessed it would be something idiotic on my side.

[lopezator]

Mine started worked today suddenly without changing anything. I guess some package updated something, added it's modules configuration or whatever...

Not sure why I was getting that error though.

[dmitris]

the reason the .git solution works is that it helps guide the go tool with information about where the VCS for custom import paths private.repo.net/user/package.git/... resides (namely (https|ssh)://private.repo.net/user/package.git)
@myitcv

If the domain contains git - for example, git.private.repo.net or foo.bar.git.company.com, would it not be reasonable for the go tool to make an educated guess that the VCS is git, and act accordingly? I would think that naming the domains for Mercurial or Bazaar with git portion in the domain would be rather pathological and extremely rare (such edge cases could be handled with a fall-back to the regular ?go-get=1 query).

[rsc]

For private.repo.net/user/package to work with go get, two things need to happen:

1. The go command has to do an HTTPS fetch of that URL and find an appropriate <meta> tag.
2. The <meta> tag has to specify an accessible git server (or module server but in this case it's git).

I don't see any indication of what meta tag is being used, exactly. If the meta tag says git:// in the git server URL (instead of https://), then you should not need any special gitconfig.

One problem might be the HTTPS fetch needing authentication. That's #26232.

Closing this issue assuming it is either a duplicate of #26232 or else solved by using a git:// URL in the meta tag.

[enugentdt]

I'm dropping into this after spending a while being extremely frustrated with the modules system.

Say I'm developing five different modules at the same time, all hosted on private GitLab. Since I'm developing them at the same time, and often, I want to work on multiple of them without committing changes, why not just use the local directory I already have cloned? A "central" list of locations of all Go modules somewhere in $GOPATH or $GOROOT would suffice. E.g. a file listing a set of "replace" directives, saying replace gitlab.com/<team>/<project>/<repo> ~/Documents/code/<project>/<repo>. It would save at least me personally one heck of a headache, and it might be a good solution for a problem that frankly was caused by Go backing itself into a corner.