sum.golang.org: go-get gitern.com/xxx/repo.git 410 gone

[urandom2]

What version of Go are you using (go version)?

$ go version
go version go1.14.6 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/user/.cache/go-build"
GOENV="/home/user/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/user/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/tmp/trash/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build897724545=/tmp/go-build -gno-record-gcc-switches"

What did you do?

go get gitern.com/urandom/errors.git

What did you expect to see?

go: downloading gitern.com/urandom/errors.git v0.0.0-20200723020514-49681dcbcc21

What did you see instead?

go: downloading gitern.com/urandom/errors.git v0.0.0-20200723020514-49681dcbcc21
go get gitern.com/urandom/errors.git@master: gitern.com/urandom/errors.git@v0.0.0-20200723020514-49681dcbcc21: verifying module: gitern.com/urandom/errors.git@v0.0.0-20200723020514-49681dcbcc21: reading https://sum.golang.org/lookup/gitern.com/urandom/errors.git@v0.0.0-20200723020514-49681dcbcc21: 410 Gone
        server response:
        not found: gitern.com/urandom/errors.git@v0.0.0-20200723020514-49681dcbcc21: invalid version: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /tmp/gopath/pkg/mod/cache/vcs/c58e8953608705577f340d5d716eb03f031f4f09a3698dde27ce667a4baf661a: exit status 128:
                fatal: unable to connect to gitern.com:
                gitern.com[0: 100.20.209.4]: errno=Connection refused

The repo appears correctly populated in ~/go/pkg/mod/cache, implying that proxy.golang.org is working, plus hitting vcs directly, via GOPRIVATE=gitern.com, works as expected.

[heschi]

You have GOPROXY="https://proxy.golang.org,direct", which means the go command will fall back to direct fetching if proxy.golang.org fails.

This is a strange one. It seems to be failing very consistently. Offhand the only thing I can think is that gitern.com is blocking the Google egress IPs that proxy.golang.org connects from. You may want to ask them if that rings any bells. I'll try to dig a little more when I get a chance.

[heschi]

cc @hyangah @katiehockman

[urandom2]

cc @huumn

[huumn]

I’m not blocking any ips explicitly. If they’re being blocked on my end they’re being blocked by AWS.

[huumn]

All repos are private on gitern which could require additional config for go get?

Edit: Ah gitern only supports git over ssh to the repos. I suspect go get uses http.

[urandom2]

Would go-get be falling back to direct for the zipball, which I have manually configured via config to use auth, then failing to find an entry in the sumdb?

[huumn]

@arnottcr gitern does not support git over http currently (only ssh). I think go get defaults to http and might need to be told to use ssh.

[urandom2]

The .git suffix causes the toolchain to try http and ssh (maybe git:// too), so i can confirm that part is working.

It would be nice to get meta header support for gitern, but that is out of scope here, and I still think there might be a go-get bug.

[hyangah]

For private repositories, please see this instruction.
https://golang.org/cmd/go/#hdr-Module_configuration_for_non_public_modules

[hyangah]

@arnottcr I just saw you already tried GOPRIVATE and it works. There is no way sum.golang.org can access private repositories (it shouldn't, right? it's private!) GOPRIVATE is the way to opting out of using proxy.golang.org and also sum.golang.org for the specified modules.

I will close this since this is working as intended.

[urandom2]

This ux is really confusing. Can we have the toolchain detect these failurse and suggest GOPRIVATE? We use it at work, and I am familiar with what it does, but it was not clear why go get was failing.

[heschi]

No, especially since I still don't know why it's failing. Connection refused isn't a reasonable error to get for a private repository; it can't possibly know whether we can authenticate without accepting the connection. My best guess is still a misconfiguration on the origin side.

At the same time, there is no point spending time diagnosing it if it's just going to fail later, so I'm not going to investigate any further.

[urandom2]

I guess my ask is when the module proxy falls back to direct, can we disable sumdb checking? I get that verification is important, but the ux is confusing, and if proxy.golang.org (or whatever module proxy you have configured) cannot reach out, why would we expect the sum.golang.org to do any better.

Since gitern may be making things more confusing, i am happy to repro with a private github repo.

[heschi]

Automatically disabling sumdb lookup would destroy the security model; any attacker that controlled the network could disable sumdb lookup by blocking access to proxy.golang.org.

Again, this is not a general problem with private modules. For example, I believe GitHub returns a warning about authentication that helps avoid confusion. The problem here is that network connectivity between proxy.golang.org and the origin is broken. There may be some room to improve the error message when sumdb fetching fails after a successful proxy download, but that would need to take into account the user's particular proxy/sumdb configuration and should definitely be discussed in a separate issue.

[urandom2]

Automatically disabling sumdb lookup would destroy the security model; any attacker that controlled the network could disable sumdb lookup by blocking access to proxy.golang.org.

If you are concerned about this why even have the ,direct option? IIUC, it will always fail, since packages that do not exist in proxy.golang.org will not have entries in sum.golang.org. If we include ,direct it needs to do something, can you walk me through a ux that does?

Furthermore, if you are concerned with network level interception, you likely have larger problems.

Again, this is not a general problem with private modules. For example, I believe GitHub returns a warning about authentication that helps avoid confusion. The problem here is that network connectivity between proxy.golang.org and the origin is broken. There may be some room to improve the error message when sumdb fetching fails after a successful proxy download, but that would need to take into account the user's particular proxy/sumdb configuration and should definitely be discussed in a separate issue.

The ux for github private repos is not better, but this may be the anti-discovery mechanism at work:

[user@localhost ~]$ go get github.com/arnottcr/private
# cd .; git clone -- https://github.com/arnottcr/private /home/user/go/src/github.com/arnottcr/private
Cloning into '/home/user/go/src/github.com/arnottcr/private'...
fatal: could not read Username for 'https://github.com': terminal prompts disabled
package github.com/arnottcr/private: exit status 128

Also the toolchain disallows the use of github.com/owner/repo.git, so it does not directly translate: (really unsure why this is disallowed)

[user@localhost ~]$ go get github.com/arnottcr/private.git
package github.com/arnottcr/private.git: invalid version control suffix in github.com/ path