sum.golang.org: negative TTL is too high

[the-maldridge]

What version of Go are you using (go version)?

$ go version
go version go1.13.4 linux/amd64

Does this issue reproduce with the latest release?

N/A

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/maldridge/.cache/go-build"
GOENV="/home/maldridge/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/maldridge/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build282271170=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I raced the release process on github foolishly believing that the Go module cache would have a short negative cache. Instead it seems that even setting GOPROXY=direct now won't let me skip the checks on sum.golang.org.

Instead I get this kind of error message:

GOPROXY=direct go get github.com/netauth/netauth@v0.3.2
verifying github.com/netauth/netauth@v0.3.2/go.mod: github.com/netauth/netauth@v0.3.2/go.mod: reading https://sum.golang.org/lookup/github.com/netauth/netauth@v0.3.2: 410 Gone

What did you expect to see?

I expected it to work, or in the worst case I expected to wait ~10 minutes and try again if the cache had a sane negative TTL.

What did you see instead?

1. GOPROXY=direct doesn't seem to actually opt out of the proxy fully. I consider this a primary bug on this issue as the documentation leads one to believe that this setting will noop the proxy, when in fact it does not.

2. I expected the cache of the 410 (Gone) to expire swiftly on a subsequent attempt. It did not I'm pretty sure it would be trivial to write a small script that just sits on a high bandwidth connection and requests non-existent version numbers of popular projects which would effectively bring releases for these projects to a halt. The fact that the negative TTL seems to be some high and undisclosed value really reduces the value of the proxy. Add in that GOPROXY=direct doesn't seem to opt out and this seems like a pretty big hindrance for the language in general.

[the-maldridge]

Additional context. I consulted this page for information: https://sum.golang.org/. Later I realized that I could use GONOSUMDB to get around the issue locally but still would need to wait for the cached failure to expire before trying to push to remote builders where this value could not be set on a per-build basis.

[toothrot]

/cc @katiehockman @heschik

[katiehockman]

GOPROXY=direct does opt out of using any proxy, including proxy.golang.org, as the documentation states. However, that is separate from sum.golang.org, which still verifies the contents of what the go command received by an origin server by default. This can be shut off using GONOSUMDB=off like you stated.

We're looking into why the error occurred though. This may be a bug that we need to address. Thanks for filing the issue.

[heschi]

As best we can tell, this was a brief connectivity problem between us and GitHub. We're improving our diagnostics and will continue to investigate any further reports, so please file a new issue if you see any more occurrences.

As for the TTL, #34370 has some background on that.

I don't think there's anything else actionable here, so closing. Please comment if you disagree.

[the-maldridge]

It would be really helpful as an end user to know if the service isn't performing nominally, as it did take me quite some time to realize that the bug wasn't in the connectivity of the build system, but rather in one of the systems my build box depended on.

[heschi]

Unfortunately there are a number of moving parts between sum.golang.org and a Git repository on GitHub, and it's not currently feasible to automatically determine the cause of a problem. Our goal is to keep the false error rate low enough that it doesn't matter.