Remove credential flag from StorageAccessProperty.CLIENT_REGION

[dimas-b]

CLIENT_REGION is not a credential value, which is in line with Iceberg's VendedCredentialsProvider code.

Cf. apache/iceberg#11389

[HonahX]

Thanks for the PR @dimas-b. To confirm the context, the s3.region was included in credentials for #342 because iceberg by that time does not support s3 cross-region access yet. That was later added in apache/iceberg@3d9fc1d and released with iceberg 1.7.0.

Would this be a breaking change for users since the s3.cross-region-access-enabled is still default to be false? I assume users who are relying on this information in vended credential to avoid cross-region call will encounter failure after this PR?

[HonahX]

Is the idea here to make the enum declaration more explicit about the isCredential setting?

[dimas-b]

Yes

[dimas-b]

@HonahX : s3.region is still included into AccessConfig and by extension into LoadTable responses, but as an ordinary property.

Is that sufficient for Iceberg clients? What's the best way to test it?

For background: the reason for this PR is that the "nice error message" from #2711 does not happen if the region is set in storage config (but no credentials are vended).

[HonahX]

@dimas-b Thanks for the pointer and context. Iceberg's LoadTableResponse have different fields holding normal config and credentials:
https://github.com/apache/iceberg/blob/03f2af8ab465d4a5238753be206bcf366130f9a0/open-api/rest-catalog-open-api.yaml#L3321-L3328

The RestCatalog client only use things in the storage-credentials fields to initialize the FileIO, where the vended credential is used.

If client.region is in extra property, it will [not be added](builder: https://github.com/apache/iceberg/blob/03f2af8ab465d4a5238753be206bcf366130f9a0/core/src/main/java/org/apache/iceberg/rest/responses/LoadTableResponse.java#L117-L130) to credential field and thus the RestCatalog client cannot pick up that config when initializing the FileIO.

[HonahX]

For background: the reason for this PR is that the "nice error message" from #2711 does not happen if the region is set in storage config (but no credentials are vended).

Would it be easier if consider the credential being "empty" if only CLIENT_REGION is there?

[dimas-b]

@HonahX : I tested with Spark 3.5 + Iceberg 1.6.1 and AWS with the code in this PR... all seems to work well... WDYT?

Note: Spark did not have any AWS FileIO config. It got everything from Polaris.

Here's a sample load table response for reference:

{
  "metadata-location": "s3://***/pol/ns/t1/metadata/00000-c8eeb778-d0ca-42ff-b48c-743f16751b7c.metadata.json",
  "metadata": {
    "format-version": 2,
    "table-uuid": "5403e77b-dbd4-425a-ba20-969bf448462b",
    "location": "s3://***/pol/ns/t1",
[...]
  },
  "config": {
    "s3.access-key-id": "***",
    "s3.secret-access-key": "****",
    "s3.session-token": "*****",
    "client.refresh-credentials-endpoint": "v1/polaris/namespaces/ns/tables/t1/credentials",
    "expiration-time": "1760987329000",
    "s3.session-token-expires-at-ms": "1760987329000",
    "client.region": "us-west-2"
  },
  "storage-credentials": [
    {
      "prefix": "s3://***/pol/ns/t1",
      "config": {
        "s3.access-key-id": "***",
        "s3.secret-access-key": "****",
        "s3.session-token": "*****",
        "expiration-time": "1760987329000",
        "s3.session-token-expires-at-ms": "1760987329000"
      }
    }
  ]
}

[HonahX]

@dimas-b Thanks for the test and examples. I also did one with Spark 3.5 + Iceberg 1.9.1. Apparently I missed some details when reading through the code. You are right! The client.region will still take effect in the config field. LGTM!