proxy.golang.org: i/o timeout frequently get i/o timeout when using bazel/gazelle

[aslonnie]

What is the URL of the page with the issue?

https://proxy.golang.org/

What is your user agent?

bazel 5.4.1 + bazel gazelle 0.33.0

Screenshot

Error looks like this:

(16:45:36) ERROR: /app/WORKSPACE:133:16: fetching go_repository rule //external:com_github_google_s2a_go: Traceback (most recent call last):
--
  | File "/root/.cache/bazel/_bazel_root/8c069df52082beee3c95ca17836fb8e2/external/bazel_gazelle/internal/go_repository.bzl", line 256, column 17, in _go_repository_impl
  | fail("failed to fetch %s: %s" % (ctx.name, result.stderr))
  | Error in fail: failed to fetch com_github_google_s2a_go: fetch_repo: github.com/google/s2a-go@v0.1.7: Get "https://proxy.golang.org/github.com/google/s2a-go/@v/v0.1.7.info": dial tcp 142.251.211.241:443: i/o timeout
  | (16:45:36) ERROR: /root/.cache/bazel/_bazel_root/8c069df52082beee3c95ca17836fb8e2/external/org_golang_google_api/internal/BUILD.bazel:3:11: @org_golang_google_api//internal:internal depends on @com_github_google_s2a_go//:s2a-go in repository @com_github_google_s2a_go which failed to fetch. no such package '@com_github_google_s2a_go//': failed to fetch com_github_google_s2a_go: fetch_repo: github.com/google/s2a-go@v0.1.7: Get "https://proxy.golang.org/github.com/google/s2a-go/@v/v0.1.7.info": dial tcp 142.251.211.241:443: i/o timeout
  | (16:45:36) ERROR: /root/.cache/bazel/_bazel_root/8c069df52082beee3c95ca17836fb8e2/external/org_golang_google_api/internal/BUILD.bazel:3:11: @org_golang_google_api//internal:internal depends on @com_github_google_s2a_go//fallback:fallback in repository @com_github_google_s2a_go which failed to fetch. no such package '@com_github_google_s2a_go//fallback': failed to fetch com_github_google_s2a_go: fetch_repo: github.com/google/s2a-go@v0.1.7: Get "https://proxy.golang.org/github.com/google/s2a-go/@v/v0.1.7.info": dial tcp 142.251.211.241:443: i/o timeout

What did you do?

Uses bazel to build our internal repository on AWS ec2 machines. We do have a lot of these build machines. They are not behind a NAT though.

What did you expect to see?

proxy.golang.org responds

What did you see instead?

proxy.golang.org timesout

it is very flaky, maybe around 1% failure (eyeballed), though when it fails, retrying does not work. it is not frequent enough to break individual workflows, but frequent enough to 100% break CI/CD pipelines with a lot of parallelism.

this issue started around Friday last week, but did not go away during the weekend.

as a mitigation, we have changed go proxy to an unofficial one + direct for now.

---

not 100% sure if this is the right place to file this bug.. seems to us more like Google's load balancer might have some issue (or we got rate limited due to some new policy?)

[cagedmantis]

cc @golang/tools-team

[findleyr]

Hi @aslonnie, can you confirm that this is still happening?

The error rate you report is inconsistent with what we see on the backend, suggesting some sort of load balancing / throttling as you suggest. I think we'd hear more loudly if this were affecting all users in this way.

[aslonnie]

seems much better now. closing. thanks.

[sfc-gh-mgalindo]

we are getting the same type of issue resulting in flakiness in our unit tests

 ERROR: /mnt/jenkins/home/jenkins/workspace/SUT-Base-Snowflake/GlobalServices/BUILD.bazel:159:19: //GlobalServices:update_build_files depends on @com_github_bazelbuild_buildtools//buildozer:buildozer in repository @com_github_bazelbuild_buildtools which failed to fetch. no such package '@com_github_bazelbuild_buildtools//buildozer': failed to fetch com_github_bazelbuild_buildtools: fetch_repo: github.com/bazelbuild/buildtools@v0.0.0-20230111132423-06e8e2436a75: Get "https://proxy.golang.org/github.com/bazelbuild/buildtools/@v/v0.0.0-20230111132423-06e8e2436a75.info": dial tcp 142.251.215.241:443: i/o timeout