feat: enable ALTS bound token (for DirectPath) in the grpc channel provider

[rockspore]

This will allow the gRPC channel provider to set up hard bound (compute engine) call credentials for the channel when DirectPath is compatible.

If the GCE/GKE metadata servers' support for such bound access tokens is not available yet, it will end up getting the normal unbound ones and nothing should fail.

No public storage API is changed.

[rockspore]

I haven't got the e2e flow verified yet but confirmed this is the change we need to enable the bound token. So I will mark this ready for review to get feedback. I will update here once the e2e flow is verified. Would you PTAL this one-liner? Thanks! @BenWhitehead

[rockspore]

Just to give some update, I opened a nearly identical googleapis/java-spanner#3645 in Cloud Spanner and was able to verify the client logic was correct internally (where the monorepo made it easier to test) by seeing the access token request Url .../token?transport=alts.

I've been trying to run the testQuickstartSampleGrpcDp in samples/snippets/.../QuickstartSampleIT.java with maximal logging but somehow couldn't find such a token url. I suspect the problem is it's not linking to the latest gax-java version (2.61.0), but haven't found out exactly why and how to resolve it.

[BenWhitehead]

When I run this branch with my dev logging I see the following request to the metadata service

2025-02-19 19:13:35,548 CONFIG   com.google.api.client.http.HttpTransport           - -------------- REQUEST  --------------
GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token
Accept-Encoding: gzip
User-Agent: Google-HTTP-Java-Client/1.46.1 (gzip)
metadata-flavor: Google
x-goog-api-client: gl-java/17.0.14 auth/1.32.1 auth-request-type/at cred-type/mds
 
2025-02-19 19:13:35,550 CONFIG   com.google.api.client.http.HttpTransport           - -------------- RESPONSE --------------
HTTP/1.1 200 OK
X-Frame-Options: SAMEORIGIN
Server: Metadata Server for VM
Content-Encoding: gzip
Metadata-Flavor: Google
X-XSS-Protection: 0
Content-Length: 869
Date: Wed, 19 Feb 2025 19:13:35 GMT
Content-Type: application/json
 
2025-02-19 19:13:35,554 CONFIG   com.google.api.client.http.HttpTransport           - Total: 1,083 bytes 
2025-02-19 19:13:35,555 CONFIG   com.google.api.client.http.HttpTransport           - {"access_token":"ya29.c.c0ASRK0GaHXA....","expires_in":2798,"token_type":"Bearer"} 

Are there other dependencies that need updated in order to see the ?transport=alts on the request?

[rockspore]

When I run this branch with my dev logging I see the following request to the metadata service

2025-02-19 19:13:35,548 CONFIG   com.google.api.client.http.HttpTransport           - -------------- REQUEST  --------------
GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token
Accept-Encoding: gzip
User-Agent: Google-HTTP-Java-Client/1.46.1 (gzip)
metadata-flavor: Google
x-goog-api-client: gl-java/17.0.14 auth/1.32.1 auth-request-type/at cred-type/mds
 
2025-02-19 19:13:35,550 CONFIG   com.google.api.client.http.HttpTransport           - -------------- RESPONSE --------------
HTTP/1.1 200 OK
X-Frame-Options: SAMEORIGIN
Server: Metadata Server for VM
Content-Encoding: gzip
Metadata-Flavor: Google
X-XSS-Protection: 0
Content-Length: 869
Date: Wed, 19 Feb 2025 19:13:35 GMT
Content-Type: application/json
 
2025-02-19 19:13:35,554 CONFIG   com.google.api.client.http.HttpTransport           - Total: 1,083 bytes 
2025-02-19 19:13:35,555 CONFIG   com.google.api.client.http.HttpTransport           - {"access_token":"ya29.c.c0ASRK0GaHXA....","expires_in":2798,"token_type":"Bearer"} 

Are there other dependencies that need updated in order to see the ?transport=alts on the request?

Thanks for testing this! Could you confirm that the gax-java dependency included googleapis/sdk-platform-java#3572? https://github.com/googleapis/sdk-platform-java/releases/tag/v2.53.0 should do. The underlying gax-grpc version should be 2.61.0.

I use VS Code locally. If I click "Go to definition" of InstantiatingGrpcChannelProvider inside GrpcStorageOptions.java, somehow it goes to 2.60 that doesn't include my PR. So I suspect it's not building with the latest version.

[BenWhitehead]

At the HEAD of your PR I see the following version of gax-grpc

[ pull/2919/head]  ❯ mvn dependency:tree -Dincludes=com.google.api:gax-grpc | grep -A 1 "com.google.cloud:google-cloud-storage:"
[INFO] com.google.cloud:google-cloud-storage:jar:2.48.3-SNAPSHOT
[INFO] \- com.google.api:gax-grpc:jar:2.61.0:compile

So, from the maven perspective it's already updated.

I ran the same code from #2919 (comment) and it produced the same request without ?transport=alts.

Do any of the auth libraries need an update as well?

[rockspore]

At the HEAD of your PR I see the following version of gax-grpc

[ pull/2919/head]  ❯ mvn dependency:tree -Dincludes=com.google.api:gax-grpc | grep -A 1 "com.google.cloud:google-cloud-storage:"
[INFO] com.google.cloud:google-cloud-storage:jar:2.48.3-SNAPSHOT
[INFO] \- com.google.api:gax-grpc:jar:2.61.0:compile

So, from the maven perspective it's already updated.

I ran the same code from #2919 (comment) and it produced the same request without ?transport=alts.

Do any of the auth libraries need an update as well?

This should be all we need because googleapis/sdk-platform-java#3572 invokes the new method from ComputeEngineCredentials and won't really build if the dependency is incorrect. This is really strange to me. I've also been trying to compile the code with a local snapshot of the auth library with additional loggings but haven't managed to let maven use the right version.

[rockspore]

I think I found the problem. Unlike in java-spanner, here the InstantiatingGrpcChannelProvider is built with being given an auth.Credentials. A ComputeEngineCredentials is required to enable DirectPath bound token before .build() is called. I think we could pass the one from the credentialsProvider here. Let me fix that quickly.

[rockspore]

I think I found the problem. Unlike in java-spanner, here the InstantiatingGrpcChannelProvider is built with being given an auth.Credentials. A ComputeEngineCredentials is required to enable DirectPath bound token before .build() is called. I think we could pass the one from the credentialsProvider here. Let me fix that quickly.

Actually it's not immediately clear to me how the credentials gets set inside the GrpcStorageOptions.java. I don't see setCredentials() of the builder being called but this invocation seems necessary. @BenWhitehead Could you help point out where the ADC gets resolved?

[BenWhitehead]

This is where we resolve the credentials instance from the base ServiceOptions

java-storage/google-cloud-storage/src/main/java/com/google/cloud/storage/GrpcStorageOptions.java

Lines 217 to 254 in 43553de

	CredentialsProvider credentialsProvider;
	Preconditions.checkState(credentials != null, "Unable to resolve credentials");
	if (credentials instanceof NoCredentials) {
	credentialsProvider = NoCredentialsProvider.create();
	} else {
	boolean foundQuotaProject = false;
	if (credentials.hasRequestMetadata()) {
	try {
	Map<String, List<String>> requestMetadata = credentials.getRequestMetadata(uri);
	for (Entry<String, List<String>> e : requestMetadata.entrySet()) {
	String key = e.getKey();
	if ("x-goog-user-project".equals(key.trim().toLowerCase(Locale.ENGLISH))) {
	List<String> value = e.getValue();
	if (!value.isEmpty()) {
	foundQuotaProject = true;
	defaultOpts = Opts.from(UnifiedOpts.userProject(value.get(0)));
	break;
	}
	}
	}
	} catch (IllegalStateException e) {
	// This happens when an instance of OAuth2Credentials attempts to refresh its
	// access token during our attempt at getting request metadata.
	// This is most easily reproduced by OAuth2Credentials.create(null);
	// see com.google.auth.oauth2.OAuth2Credentials.refreshAccessToken
	if (!e.getMessage().startsWith("OAuth2Credentials")) {
	throw e;
	}
	}
	}
	if (foundQuotaProject) {
	// fix for https://github.com/googleapis/java-storage/issues/1736
	credentialsProvider =
	FixedCredentialsProvider.create(new QuotaProjectIdHidingCredentials(credentials));
	} else {
	credentialsProvider = FixedCredentialsProvider.create(credentials);
	}
	}

And here is where we pass it on to the gapic client builder:

java-storage/google-cloud-storage/src/main/java/com/google/cloud/storage/GrpcStorageOptions.java

Line 275 in 43553de

.setCredentialsProvider(credentialsProvider)

[rockspore]

Thank you so much for your help so far, @BenWhitehead! Your maven knowledge saved me tons of time! Now I was able to confirm that with the current change, it will invoke the correct logic in InstantiatingGrpcChannelProvider to build the hard bound credentials.

The issue seems to be in how the hard bound credentials got created: https://github.com/googleapis/sdk-platform-java/blob/ad26cf98548e325c99edb263baf8fe1a7696e634/gax-java/gax-grpc/src/main/java/com/google/api/gax/grpc/InstantiatingGrpcChannelProvider.java#L1205-L1209. It converts the existing creds into a builder which actually copies over the accessToken class member of the parent OAuth2Credentials: https://github.com/googleapis/google-auth-library-java/blob/a8fe21f3ed6416c8c5be02edc0a22234e4eddad7/oauth2_http/java/com/google/auth/oauth2/OAuth2Credentials.java#L657. So since a token request without ?transport=alts had been made before, the new credentials never bothered getting the token again.

In the CloudSpanner examples maybe the token hadn't been populated yet, so I was able to see the request with ?transport=alts.

In summary, this PR itself should have no problem. @rmehta19 and I will work on fixing the logic in gax. I think this PR can wait till gax is fixed and the dependencies get updated.

[BenWhitehead]

Sounds good @rockspore, please let me know when you want me to consider merging this.

[rockspore]

Hi @BenWhitehead to give you an update. Technically this PR is ready to be merged. But as I talked within the team, because GCS already has DirectPath as a GA feature, we would like to play safe here. Meanwhile, I also feel reluctant to add a new flag guard to toggle this because java-storage doesn't have known internal users and a default-off pretty much means no adoption.

So my plan is to wait till we get some good signal from at least one of the following:

1. Cloud Spanner.
2. storage-go (The feature is waiting for the guac-go PR and I'm in conversation with some other GCS folks).

before we merge this PR. Let me know if you have a better idea. Thanks.

[BenWhitehead]

a default-off pretty much means no adoption

I tend to agree. Holding off until we have signal from a different bundle I think makes sense.