Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

storage: timeout for /computeMetadata/v1/universe/universe_domain in storage.NewClient #9350

Closed
jaimem88 opened this issue Feb 1, 2024 · 10 comments · Fixed by googleapis/google-api-go-client#2393
Closed

storage: timeout for /computeMetadata/v1/universe/universe_domain in storage.NewClient #9350

jaimem88 opened this issue Feb 1, 2024 · 10 comments · Fixed by googleapis/google-api-go-client#2393
Assignees
@quartzmo @BrennaEpp
Labels
api: storage Issues related to the Cloud Storage API. priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@jaimem88
Copy link

jaimem88 commented Feb 1, 2024

Environment details

  • Programming language: Go
  • OS: docker image golang:1.21
  • Language runtime version: go1.21.6
  • Package version: v0.161.0

Steps to reproduce

The latest update is breaking our pipeline when we try to update the library:
https://gitlab.com/gitlab-org/container-registry/-/merge_requests/1569.

One of our integration tests keeps failing as seen in this CI job.

panic: Error creating storage client: dialing: Get "http://169.254.169.254/computeMetadata/v1/universe/universe_domain": dial tcp 169.254.169.254:80: i/o timeout
goroutine 1 [running]:
github.com/docker/distribution/registry/storage/driver/gcs.init.1()
	/builds/gitlab-org/container-registry/registry/storage/driver/gcs/gcs_test.go:87 +0x8c8
FAIL	github.com/docker/distribution/registry/storage/driver/gcs	12.910s
=== Failed
=== FAIL: registry/storage/driver/gcs  (0.00s)
panic: Error creating storage client: dialing: Get "http://169.254.169.254/computeMetadata/v1/universe/universe_domain": dial tcp 169.254.169.254:80: i/o timeout
goroutine 1 [running]:
github.com/docker/distribution/registry/storage/driver/gcs.init.1()
	/builds/gitlab-org/container-registry/registry/storage/driver/gcs/gcs_test.go:87 +0x8c8
FAIL	github.com/docker/distribution/registry/storage/driver/gcs	12.910s
DONE 0 tests, 1 failure in 81.895s
exit status 1

This seems suspiciously related to googleapis/google-api-go-client@b21a1fa. I am not sure where the hostname is coming from but it seems that the DefaultEndpoint is not being set properly.
@jaimem88 jaimem88 added priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. labels Feb 1, 2024
@codyoss codyoss assigned quartzmo and unassigned codyoss Feb 1, 2024
@quartzmo
Copy link
Member

quartzmo commented Feb 1, 2024

@jaimem88 Thank you for reporting this bug. I will try to reproduce it today.

@quartzmo quartzmo transferred this issue from googleapis/google-api-go-client Feb 1, 2024
@quartzmo quartzmo added the api: storage Issues related to the Cloud Storage API. label Feb 1, 2024
@quartzmo
Copy link
Member

quartzmo commented Feb 1, 2024

@jaimem88 I have been unable to reproduce it so far.

The request that times out according to your logs is to the Compute Metadata Server. It should only be made if the default credentials are determined to be provided by the Compute Metadata Server. Just from static inspection (ie, looking) the bug appears to be in this line.

@quartzmo
Copy link
Member

quartzmo commented Feb 1, 2024

Do you know if your CI runs on GCE and if so, exactly which GCE-based product? (GKE, Cloud Run?). It seems metadata.OnGCE() is returning true but the Metadata Server endpoint is timing out.

@quartzmo
Copy link
Member

quartzmo commented Feb 1, 2024

Can you provide the full go.mod settings?

@jaimem88
Copy link
Author

jaimem88 commented Feb 1, 2024

Thanks for looking into this @quartzmo!

Can you provide the full go.mod settings?

Here are the full go.mod and go.sum files.

module github.com/docker/distribution

go 1.18

require (
	cloud.google.com/go/storage v1.37.0
	github.com/Azure/azure-sdk-for-go v68.0.0+incompatible
	github.com/DATA-DOG/go-sqlmock v1.5.2
	github.com/Shopify/toxiproxy/v2 v2.7.0
	github.com/alicebob/miniredis/v2 v2.31.1
	github.com/aws/aws-sdk-go v1.46.7
	github.com/benbjohnson/clock v1.3.5
	github.com/cenkalti/backoff/v4 v4.2.1
	github.com/denverdino/aliyungo v0.0.0-20230411124812-ab98a9173ace
	github.com/docker/go-metrics v0.0.1
	github.com/docker/libtrust v0.0.0-20160708172513-aabc10ec26b7
	github.com/eko/gocache/lib/v4 v4.1.5
	github.com/eko/gocache/store/redis/v4 v4.2.0
	github.com/getsentry/sentry-go v0.26.0
	github.com/go-redis/redismock/v9 v9.0.3
	github.com/golang/mock v1.6.0
	github.com/gorilla/handlers v1.5.2
	github.com/gorilla/mux v1.8.1
	github.com/hashicorp/go-multierror v1.1.1
	github.com/jackc/pgerrcode v0.0.0-20220416144525-469b46aa5efa
	github.com/jackc/pgx/v5 v5.5.2
	github.com/jszwec/csvutil v1.9.0
	github.com/mitchellh/mapstructure v1.5.0
	github.com/ncw/swift v1.0.53
	github.com/olekukonko/tablewriter v0.0.5
	github.com/opencontainers/go-digest v1.0.0
	github.com/opencontainers/image-spec v1.0.2
	github.com/prometheus/client_golang v1.18.0
	github.com/redis/go-redis/v9 v9.4.0
	github.com/rubenv/sql-migrate v1.5.2
	github.com/schollz/progressbar/v3 v3.14.1
	github.com/sirupsen/logrus v1.9.3
	github.com/spf13/cobra v1.8.0
	github.com/spf13/viper v1.18.2
	github.com/stretchr/testify v1.8.4
	github.com/trim21/go-redis-prometheus v0.0.0
	github.com/vmihailenco/msgpack/v5 v5.4.1
	github.com/xanzy/go-gitlab v0.96.0
	gitlab.com/gitlab-org/labkit v1.21.0
	go.uber.org/automaxprocs v1.5.3
	golang.org/x/crypto v0.18.0
	golang.org/x/oauth2 v0.16.0
	golang.org/x/sync v0.6.0
	golang.org/x/time v0.5.0
	google.golang.org/api v0.161.0
	gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c
	gopkg.in/yaml.v2 v2.4.0
)

require (
	cloud.google.com/go v0.112.0 // indirect
	cloud.google.com/go/compute v1.23.3 // indirect
	cloud.google.com/go/compute/metadata v0.2.3 // indirect
	cloud.google.com/go/iam v1.1.5 // indirect
	cloud.google.com/go/profiler v0.1.0 // indirect
	github.com/Azure/go-autorest v14.2.0+incompatible // indirect
	github.com/Azure/go-autorest/autorest v0.11.29 // indirect
	github.com/Azure/go-autorest/autorest/adal v0.9.23 // indirect
	github.com/Azure/go-autorest/autorest/date v0.3.0 // indirect
	github.com/Azure/go-autorest/autorest/to v0.4.0 // indirect
	github.com/Azure/go-autorest/logger v0.2.1 // indirect
	github.com/Azure/go-autorest/tracing v0.6.0 // indirect
	github.com/alicebob/gopher-json v0.0.0-20200520072559-a9ecdc9d1d3a // indirect
	github.com/beorn7/perks v1.0.1 // indirect
	github.com/cespare/xxhash/v2 v2.2.0 // indirect
	github.com/client9/reopen v1.0.0 // indirect
	github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
	github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
	github.com/dnaeon/go-vcr v1.0.1 // indirect
	github.com/felixge/httpsnoop v1.0.4 // indirect
	github.com/fsnotify/fsnotify v1.7.0 // indirect
	github.com/go-gorp/gorp/v3 v3.1.0 // indirect
	github.com/go-logr/logr v1.4.1 // indirect
	github.com/go-logr/stdr v1.2.2 // indirect
	github.com/gofrs/uuid v4.4.0+incompatible // indirect
	github.com/golang-jwt/jwt/v4 v4.5.0 // indirect
	github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
	github.com/golang/protobuf v1.5.3 // indirect
	github.com/google/go-querystring v1.1.0 // indirect
	github.com/google/pprof v0.0.0-20210804190019-f964ff605595 // indirect
	github.com/google/s2a-go v0.1.7 // indirect
	github.com/google/uuid v1.5.0 // indirect
	github.com/googleapis/enterprise-certificate-proxy v0.3.2 // indirect
	github.com/googleapis/gax-go/v2 v2.12.0 // indirect
	github.com/hashicorp/errwrap v1.1.0 // indirect
	github.com/hashicorp/go-cleanhttp v0.5.2 // indirect
	github.com/hashicorp/go-retryablehttp v0.7.2 // indirect
	github.com/hashicorp/hcl v1.0.0 // indirect
	github.com/inconshreveable/mousetrap v1.1.0 // indirect
	github.com/jackc/pgpassfile v1.0.0 // indirect
	github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a // indirect
	github.com/jackc/puddle/v2 v2.2.1 // indirect
	github.com/jmespath/go-jmespath v0.4.0 // indirect
	github.com/kr/pretty v0.3.1 // indirect
	github.com/kr/text v0.2.0 // indirect
	github.com/magiconair/properties v1.8.7 // indirect
	github.com/mattn/go-runewidth v0.0.12 // indirect
	github.com/matttproud/golang_protobuf_extensions/v2 v2.0.0 // indirect
	github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db // indirect
	github.com/oklog/ulid/v2 v2.0.2 // indirect
	github.com/opentracing/opentracing-go v1.2.0 // indirect
	github.com/pelletier/go-toml/v2 v2.1.0 // indirect
	github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
	github.com/prometheus/client_model v0.5.0 // indirect
	github.com/prometheus/common v0.45.0 // indirect
	github.com/prometheus/procfs v0.12.0 // indirect
	github.com/rivo/uniseg v0.4.4 // indirect
	github.com/rogpeppe/go-internal v1.10.0 // indirect
	github.com/sagikazarmark/locafero v0.4.0 // indirect
	github.com/sagikazarmark/slog-shim v0.1.0 // indirect
	github.com/sebest/xff v0.0.0-20210106013422-671bd2870b3a // indirect
	github.com/sourcegraph/conc v0.3.0 // indirect
	github.com/spf13/afero v1.11.0 // indirect
	github.com/spf13/cast v1.6.0 // indirect
	github.com/spf13/pflag v1.0.5 // indirect
	github.com/subosito/gotenv v1.6.0 // indirect
	github.com/vmihailenco/tagparser/v2 v2.0.0 // indirect
	github.com/yuin/gopher-lua v1.1.0 // indirect
	go.opencensus.io v0.24.0 // indirect
	go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.47.0 // indirect
	go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.47.0 // indirect
	go.opentelemetry.io/otel v1.22.0 // indirect
	go.opentelemetry.io/otel/metric v1.22.0 // indirect
	go.opentelemetry.io/otel/trace v1.22.0 // indirect
	go.uber.org/atomic v1.9.0 // indirect
	go.uber.org/multierr v1.9.0 // indirect
	golang.org/x/exp v0.0.0-20230905200255-921286631fa9 // indirect
	golang.org/x/net v0.20.0 // indirect
	golang.org/x/sys v0.16.0 // indirect
	golang.org/x/term v0.16.0 // indirect
	golang.org/x/text v0.14.0 // indirect
	google.golang.org/appengine v1.6.8 // indirect
	google.golang.org/genproto v0.0.0-20240116215550-a9fa1716bcac // indirect
	google.golang.org/genproto/googleapis/api v0.0.0-20240122161410-6c6643bf1457 // indirect
	google.golang.org/genproto/googleapis/rpc v0.0.0-20240116215550-a9fa1716bcac // indirect
	google.golang.org/grpc v1.60.1 // indirect
	google.golang.org/protobuf v1.32.0 // indirect
	gopkg.in/ini.v1 v1.67.0 // indirect
	gopkg.in/yaml.v3 v3.0.1 // indirect
)

Do you know if your CI runs on GCE and if so, exactly which GCE-based product? (GKE, Cloud Run?). It seems metadata.OnGCE() is returning true but the Metadata Server endpoint is timing out.

I believe our gitlab-runner runs on GKE, and the job is executed from a golang:1.21 container.

Just for reference, the same job on the previous version v0.157.0 of the library runs fine https://gitlab.com/gitlab-org/container-registry/-/jobs/6058207860.

@quartzmo quartzmo changed the title dialing: Get "http://169.254.169.254/computeMetadata/v1/universe/universe_domain" when initializing storage client storage: timeout for /computeMetadata/v1/universe/universe_domain in storage.NewClient Feb 1, 2024
@quartzmo
Copy link
Member

quartzmo commented Feb 2, 2024 •
edited
Loading

@jaimem88 Are you still seeing this issue? If so, would it be possible for you to run the following curl commands in your GKE environment in order to test if the failing request is transient or consistent?

The commands are almost identical, the difference is just IP address vs domain name. As you can see from the IP address, the URL in your logs is correct. It is not supposed to timeout, but the rollout of this feature seems to possibly be inconsistent. Otoh, the timeout might have been a rare transient issue such as a restart. Thank you!

curl -H "Metadata-Flavor:Google"  http://169.254.169.254/computeMetadata/v1/universe/universe_domain

curl -H "Metadata-Flavor:Google" http://metadata.google.internal/computeMetadata/v1/universe/universe_domain

quartzmo added a commit to quartzmo/google-api-go-client that referenced this issue Feb 3, 2024
@quartzmo
fix(transport): enforce 1s timeout on requests to MDS universe_domain
c31f3a1 
* Temporarily return googleapis.com on MDS universe_domain timeout.

closes: googleapis/google-cloud-go#9350
This was referenced Feb 3, 2024
Merged
Closed
quartzmo added a commit to googleapis/google-api-go-client that referenced this issue Feb 5, 2024
@quartzmo
fix(transport): enforce 1s timeout on requests to MDS universe_domain (…
6862015 
…#2393)

* Temporarily return googleapis.com on MDS universe_domain timeout.

closes: googleapis/google-cloud-go#9350
@quartzmo
Copy link
Member

quartzmo commented Feb 6, 2024

@jaimem88 Please upgrade to https://github.com/googleapis/google-api-go-client/releases/tag/v0.162.0 if possible, and let us know if the error is resolved.

@jaimem88
Copy link
Author

jaimem88 commented Feb 6, 2024

Thank you @quartzmo! The latest version works as before https://gitlab.com/gitlab-org/container-registry/-/jobs/6111764468 🎉

@clerk-zach
Copy link

We are experiencing this issue upgrading logging from 0.7.0 to 0.9.0

@codyoss
Copy link
Member

codyoss commented Feb 29, 2024

@clerk-zach Did you try the above workaround?

@JMTK JMTK mentioned this issue Jul 8, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@quartzmo quartzmo

@BrennaEpp BrennaEpp

Labels
api: storage Issues related to the Cloud Storage API. priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(transport): enforce 1s timeout on requests to MDS universe_domain quartzmo/google-api-go-client
5 participants
@quartzmo @codyoss @jaimem88 @BrennaEpp @clerk-zach