proxy.golang.org: widespread not found: fetch timed out

[JoshuaSjoding]

I pushed an untagged commit to the github.com/scjalliance/astconf module 3 hours ago, but sum.golang.org claims that it can't find it when requesting its pseudo-version v0.0.0-20221209034541-5c12b5578696. It returns a 404 Not Found result.

Running go list -m for the branch does list the expected commit, but takes a very long time to run:

time go list -m -json github.com/scjalliance/astconf@main
{
        "Path": "github.com/scjalliance/astconf",
        "Version": "v0.0.0-20221209034541-5c12b5578696",
        "Query": "main",
        "Time": "2022-12-09T03:45:41Z",
        "Origin": {
                "VCS": "git",
                "URL": "https://github.com/scjalliance/astconf",
                "TagSum": "t1:47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=",
                "Ref": "refs/heads/main",
                "Hash": "5c12b5578696a1517acd34d3c60d88b303b0b964"
        }
}

real    2m55.660s
user    0m0.115s
sys     0m0.068s

Performing a go get for the branch returns server response: not found: fetch timed out:

go get github.com/scjalliance/astconf@main
go: downloading github.com/scjalliance/astconf v0.0.0-20221209034541-5c12b5578696
go: github.com/scjalliance/astconf@main: github.com/scjalliance/astconf@v0.0.0-20221209034541-5c12b5578696: verifying module: github.com/scjalliance/astconf@v0.0.0-20221209034541-5c12b5578696: reading https://sum.golang.org/lookup/github.com/scjalliance/astconf@v0.0.0-20221209034541-5c12b5578696: 404 Not Found
        server response: not found: fetch timed out

I get a not found: fetch timed out response after waiting 58 seconds when I go here:

• https://sum.golang.org/lookup/github.com/scjalliance/astconf@v0.0.0-20221209034541-5c12b5578696

I don't know if it matters, but I did perform a forced git push to the astconf repository earlier in the evening, and I renamed the branch from master to main yesterday.

I should also note that pkg.go.dev lists v0.0.0-20221209023833-cad60e1e1338 as the latest version right now, but when viewing that version specifically it also claims that it isn't the latest version. I used the package request feature of pkg.go.dev to request the v0.0.0-20221209023833-cad60e1e1338 release earlier this evening.

I've been watching the GitHub Status Page and it claims all is well.

I am filing this report as directed by the FAQ on sum.golang.org, which was added in response to #42809:

Note that if someone requested the version before the tag was pushed, it may take up to 30 minutes for the mirror's cache to expire and fresh data about the version to become available. If the version is still not available after 30 minutes, please file an issue.

I first encountered this issue while running the Go 1.19.2 tooling on windows/amd64. I updated to 1.19.4 but the issue persists. The slow response to go list -m was reproduced in a linux environment.

[jclapis]

I ran into this as well today; I was able to get things moving again with export GOSUMDB=off and export GOPROXY=direct for now.

[hajimehoshi]

Getting the latest version of a binary via the Go proxy also fails:

$ go install -x -v golang.org/x/mobile/cmd/gobind@latest
# get https://proxy.golang.org/golang.org/x/mobile/@v/list
# get https://proxy.golang.org/golang.org/x/mobile/cmd/gobind/@v/list
# get https://proxy.golang.org/golang.org/x/mobile/cmd/@v/list
# get https://proxy.golang.org/golang.org/x/@v/list
# get https://proxy.golang.org/golang.org/@v/list
# get https://proxy.golang.org/golang.org/@v/list: 404 Not Found (0.121s)
# get https://proxy.golang.org/golang.org/x/@v/list: 404 Not Found (0.118s)
# get https://proxy.golang.org/golang.org/x/mobile/cmd/@v/list: 404 Not Found (58.321s)
# get https://proxy.golang.org/golang.org/x/mobile/@v/list: 200 OK (58.325s)
# get https://proxy.golang.org/golang.org/x/mobile/cmd/gobind/@v/list: 404 Not Found (58.322s)
# get https://proxy.golang.org/golang.org/x/mobile/@latest
# get https://proxy.golang.org/golang.org/x/mobile/@latest: 404 Not Found (58.048s)
go: golang.org/x/mobile/cmd/gobind@latest: no matching versions for query "latest"

[afumu]

go env -w GOPROXY=direct
go env -w GOSUMDB=off

[VetSoftKeith]

Also experiencing this on v1.19.4 (rolled back to 1.19.3 because I thought it might have been update related at first). I got a notification of a timed out pipeline from GitLab (it's a 1h timeout). The pipeline was stuck when attempting to fetch external packages with go get -u ./.... Couldn't get it to run on Linux (Ubuntu in Docker) or Windows 10/11. @zouchangfu's workaround works good for now but I hate putting workarounds into production just so I can deploy on schedule

[seankhliao]

cc @golang/tools-team

[oiooj]

There’s a tmp public sumdb server (maybe the only one outsie google): https://gosum.io

[seankhliao]

please use the emoji voting on the issue if you're seeing the issue, we don't need a hundred comments of the same

[prattmic]

Hi all, I'm starting to look into this. Not sure what is going on yet.

[wallyqs]

In our case, we have a repo that does not prepend ‘v’ to the tag version due to historical reasons and go install to a pinned version now takes very long:

go install -x -n -a mailto:github.com/nats-io/nsc@2.7.5
# get https://proxy.golang.org/github.com/@v/2.7.5.info
# get https://proxy.golang.org/github.com/nats-io/@v/2.7.5.info
# get https://proxy.golang.org/github.com/nats-io/nsc/@v/2.7.5.info
# get https://proxy.golang.org/github.com/nats-io/@v/2.7.5.info: 404 Not Found (0.272s)
# get https://proxy.golang.org/github.com/@v/2.7.5.info: 404 Not Found (0.272s)
# get https://proxy.golang.org/github.com/nats-io/nsc/@v/2.7.5.info: 404 Not Found (58.331s)
# ( from here on it does fetch the repo and eventually works)

also fetch latest stopped working:

go install -x -n -a http://github.com/nats-io/nsc@latest
# get https://proxy.golang.org/github.com/@v/list
# get https://proxy.golang.org/github.com/nats-io/nsc/@v/list
# get https://proxy.golang.org/github.com/nats-io/@v/list
# get https://proxy.golang.org/github.com/@v/list: 404 Not Found (0.055s)
# get https://proxy.golang.org/github.com/nats-io/@v/list: 404 Not Found (0.056s)
# get https://proxy.golang.org/github.com/nats-io/nsc/@v/list: 200 OK (58.077s)
# get https://proxy.golang.org/github.com/nats-io/nsc/@latest
# get https://proxy.golang.org/github.com/nats-io/nsc/@latest: 404 Not Found (58.080s)
go: http://github.com/nats-io/nsc@latest: no matching versions for query "latest"

[lallydd]

Part of the problem may have been a transient networking issue. For some time I had 50% packet loss to proxy.golang.org that subsided 3 hours ago.

[prattmic]

We believe that the service has recovered and fetch/list should be succeeding again. There is still a tail of background refreshes completing, so some fetches may still take longer than normal because they are not cached.

[zbindenren]

I can confirm that it works for us again.

[wallyqs]

Thanks @prattmic! It is working for us too now. Is it possible to share a quick summary on what caused the issue? Also curious where is the 1 minute get timeout coming from?

[JoshuaSjoding]

I ran go list -m -json github.com/scjalliance/astconf@main again and can confirm that it completed in about 5 seconds, which is great.

I was able to perform go get github.com/scjalliance/astconf@main and it completed successfully in about 10 seconds, which is also great.

The pkg.go.dev representation of astconf still seems to be confused about what the latest version is, but maybe that's another issue.

[prattmic]

@wallyqs We received a large amount of distributed traffic overnight for module versions that did not exist. This led to a backlog of versions to process that we've only just burned down. The 1 minute timeout is inside of proxy.golang.org.

[prattmic]

I'm closing this as the fetching issues should be resolved. Please open another bug if you are experience more issues.

[nikrozman21]

This issue just resurfaced for me.

[benhoyt]

This is happening again for me now (last comment was 2 days ago, so this might be another incident).

[prattmic]

We have continued to have a couple of additional sporadic issues in the same vain as the original issue, though shorter duration. We are still working to mitigate.

[prattmic]

We have rolled out additional mitigations and have not seen any issues in around a week.

[ValarDragon]

I'm sporadically getting the same issue right now, e.g. with: github.com/osmosis-labs/osmosis/osmoutils@v0.0.1

• link: https://sum.golang.org/lookup/github.com/osmosis-labs/osmosis/osmoutils@v0.0.1
• git tag: https://github.com/osmosis-labs/osmosis/releases/tag/osmoutils%2Fv0.0.1

(12 minutes)

[hyangah]

@ValarDragon Do you see issues with other modules, or only those in github.com/osmosis-labs/osmosis/* ?

The module version was published around 2022-12-22T04:27:37 UTC.
https://proxy.golang.org/github.com/osmosis-labs/osmosis/osmoutils/@v/v0.0.1.info

It looks like someone requested it slightly before 2022-12-21T04:27:16 UTC and proxy.golang.org failed to see the release. As being discussed in #42809, proxy.golang.org retried later (~30min) and found the version in the origin. I think this is a duplicate of #42809, rather than this issue.

From the first FAQ entry of proxy.golang.org:

I committed a new change (or released a new version) to a repository, why isn't it showing up when I run go get -u or go list -m --versions?
In order to improve our services' caching and serving latencies, new versions may not show up right away. If you want new code to be immediately available in the mirror, then first make sure there is a semantically versioned tag for this revision in the underlying source repository. Then explicitly request that version via go get module@version. The new version should be available within one minute. Note that if someone requested the version before the tag was pushed, it may take up to 30 minutes for the mirror's cache to expire and fresh data about the version to become available.

[ValarDragon]

Ohh I think your totally right, thank you!