Add `pathStyleAccess` to AwsStorageConfigInfo #2012

dimas-b · 2025-07-08T23:46:48Z

This change allows configuring the "path-style" access mode in S3 clients (both in Polaris Servers and Iceberg REST Catalog API clients).

This change is applicable both to AWS storage and to non-AWS S3-compatible storage (#1530).

The Management API change is backward-compatible with clients lenient to unknown JSON properties.

Includes a fix to AWS config object round-trip via generated API classes.

Dev ML discussion: https://lists.apache.org/thread/oxy2yd4mbq06zr4z57lfptgy95c3lyhb

polaris-core/src/main/java/org/apache/polaris/core/storage/aws/AwsStorageConfigurationInfo.java

eric-maynard · 2025-07-09T03:03:10Z

polaris-core/src/main/java/org/apache/polaris/core/storage/aws/AwsStorageConfigurationInfo.java

We could use the default here, or just leave it null to leverage the default used in orElse. Better not to proliferate the constants around like this.

changed to use PATH_STYLE_ACCESS_DEFAULT

Now that we have a default at the API level, is this actually nullable?

I had to remove the API default.

spec/polaris-management-service.yml

polaris-core/src/main/java/org/apache/polaris/core/storage/aws/AwsStorageConfigurationInfo.java

singhpk234

LGTM !

dimas-b · 2025-07-09T14:44:58Z

I had to remove one commit (discussed in the API yaml comment thread). PTAL.

eric-maynard

Let's clarify if this is a breaking change (does it need to target 2.0? Go into API version 2?) and plan accordingly before merging.

snazy

LGTM

dimas-b · 2025-07-14T17:10:12Z

@eric-maynard : I believe I addressed your backward-compatibility concern in the latest push. Could you review again?

This change allows configuring the "path-style" access mode in S3 clients (both in Polaris Servers and Iceberg REST Catalog API clients). This change is applicable both to AWS storage and to non-AWS S3-compatible storage (apache#1530). The Management API change is backward-compatible.

eric-maynard · 2025-07-16T03:59:12Z

spec/polaris-management-service.yml

+              type: boolean
+              description: >- 
+                Whether S3 requests to files in this catalog should use 'path-style addressing for buckets'.
+                Default: false.


nit: With the default in the spec, I don't think we need the default in the doc. Other similar variables don't have it.

removed 👍

Amends apache#1913 and apache#2012

…ess (#2127) This change adds support for endpoint, sts-endpoint, path-style-access to the Polaris Python client. Amends #1913 and #2012

* Add `pathStyleAccess` to AwsStorageConfigInfo This change allows configuring the "path-style" access mode in S3 clients (both in Polaris Servers and Iceberg REST Catalog API clients). This change is applicable both to AWS storage and to non-AWS S3-compatible storage (apache#1530). (cherry picked from commit 987c554)

* chore(deps): update dependency mypy to >=1.17, <=1.17.0 (apache#2114) * Spark 3.5.6 and Iceberg 1.9.1 (apache#1960) * Spark 3.5.6 and Iceberg 1.9.1 * Cleanup * Add `pathStyleAccess` to AwsStorageConfigInfo (apache#2012) * Add `pathStyleAccess` to AwsStorageConfigInfo This change allows configuring the "path-style" access mode in S3 clients (both in Polaris Servers and Iceberg REST Catalog API clients). This change is applicable both to AWS storage and to non-AWS S3-compatible storage (apache#1530). * Add TestFileIOFactory helper (apache#2105) * Add FileIOFactory.wrapExisting helper * fix(deps): update dependency gradle.plugin.org.jetbrains.gradle.plugin.idea-ext:gradle-idea-ext to v1.2 (apache#2125) * fix(deps): update dependency boto3 to v1.39.7 (apache#2124) * Abstract polaris-runtime-service tests for all persistence implementations (apache#2106) The NoSQL persistence implementation has to run the Iceberg table & view catalog plus the Polaris specific tests as well. Reusing existing tests is beneficial to avoid a lot of code duplcation. This change moves the actual tests to `Abstract*` classes and refactors the existing tests to extend those. The NoSQL persistence work extends the same `Abstract*` classes but runs with different Quarkus test profiles. * Add IMPLICIT authentication support to the CLI (apache#2121) PRs apache#1925 and apache#1912 were merged around the same time. This PR connects the two changes and enables the CLI to accept IMPLICIT authentication type. Since Hadoop federated catalogs rely purely on IMPLICIT authentication, the CLI parsing test has been updated to reflect the same. * feat(helm): Add support for external authentication (apache#2104) * fix(deps): update dependency org.apache.iceberg:iceberg-bom to v1.9.2 (apache#2126) * fix(deps): update quarkus platform and group to v3.24.4 (apache#2128) * fix(deps): update dependency boto3 to v1.39.8 (apache#2129) * fix(deps): update dependency io.smallrye.config:smallrye-config-core to v3.13.3 (apache#2130) * Add newIcebergCatalog helper (apache#2134) creation of `IcebergCatalog` instances was quite redundant as tests mostly use the same parameters most of the time. also remove an unused field in 2 other tests. * Add server and client support for the new generic table `baseLocation` field (apache#2122) * Use Makefile to simplify setup and commands (apache#2027) * Use Makefile to simplify setup and commands * Add targets for minikube state management * Add podman support and spark plugin build * Add version target * Update README.md for Makefile usage and relation to the project * Fix nit * Package polaris client as python package (apache#2049) * Package polaris client as python package * Package polaris client as python package * Change owner to spark when copying files from local into Dockerfile * CI: Address failure from accessing GH API (apache#2132) CI sometimes fails with this failure: ``` * What went wrong: Execution failed for task ':generatePomFileForMavenPublication'. > Unable to process url: https://api.github.com/repos/apache/polaris/contributors?per_page=1000 ``` The sometimes failing request fetches the list of contributors to be published in the "root" POM. Unauthorized GH API requests have an hourly(?) limit of 60 requests per source IP. Authorized requests have a much higher rate limit. We do have a GitHub token available in every CI run, which can be used in GH API requests. This change adds the `Authorization` header for the failing GH API request to leverage the higher rate limit and let CI not fail (that often). * fix(deps): update dependency com.nimbusds:nimbus-jose-jwt to v10.4 (apache#2139) * fix(deps): update dependency com.diffplug.spotless:spotless-plugin-gradle to v7.2.0 (apache#2142) * fix(deps): update dependency software.amazon.awssdk:bom to v2.32.4 (apache#2146) * fix(deps): update dependency org.xerial.snappy:snappy-java to v1.1.10.8 (apache#2138) * fix(deps): update dependency org.junit:junit-bom to v5.13.4 (apache#2147) * fix(deps): update dependency boto3 to v1.39.9 (apache#2137) * fix(deps): update dependency com.fasterxml.jackson:jackson-bom to v2.19.2 (apache#2136) * Python client: add support for endpoint, sts-endpoint, path-style-access (apache#2127) This change adds support for endpoint, sts-endpoint, path-style-access to the Polaris Python client. Amends apache#1913 and apache#2012 * Remove PolarisEntityManager.getCredentialCache (apache#2133) `PolarisEntityManager` itself is not using the `StorageCredentialCache` but just hands it out via `getCredentialCache`. the only caller of `getCredentialCache` is `FileIOUtil.refreshAccessConfig`, which in in turn is only called by `DefaultFileIOFactory` and `IcebergCatalog`. note that in a follow-up we will likely be able to remove `PolarisEntityManager` usage completely from `IcebergCatalog`. additional cleanups: - use `StorageCredentialCache` injection in tests (but we need to invalidate all entries on test start) - remove unused `UserSecretsManagerFactory` from `PolarisCallContextCatalogFactory` * chore(deps): update registry.access.redhat.com/ubi9/openjdk-21-runtime docker tag to v1.22-1.1752676419 (apache#2150) * fix(deps): update dependency com.diffplug.spotless:spotless-plugin-gradle to v7.2.1 (apache#2152) * fix(deps): update dependency boto3 to v1.39.10 (apache#2151) * chore: fix class reference in the javadoc of TableLikeEntity (apache#2157) * fix(deps): update dependency commons-codec:commons-codec to v1.19.0 (apache#2160) * fix(deps): update dependency boto3 to v1.39.11 (apache#2159) * Last merged commit 395459f --------- Co-authored-by: Mend Renovate <bot@renovateapp.com> Co-authored-by: Yong Zheng <yongzheng0809@gmail.com> Co-authored-by: Dmitri Bourlatchkov <dmitri.bourlatchkov@gmail.com> Co-authored-by: Christopher Lambert <xn137@gmx.de> Co-authored-by: Pooja Nilangekar <poojan@umd.edu> Co-authored-by: Alexandre Dutra <adutra@apache.org> Co-authored-by: Yun Zou <yunzou.colostate@gmail.com>

dimas-b requested review from eric-maynard and snazy July 8, 2025 23:46

github-project-automation bot added this to Basic Kanban Board Jul 8, 2025

github-project-automation bot moved this to PRs In Progress in Basic Kanban Board Jul 8, 2025

eric-maynard reviewed Jul 9, 2025

View reviewed changes

polaris-core/src/main/java/org/apache/polaris/core/storage/aws/AwsStorageConfigurationInfo.java Outdated Show resolved Hide resolved

eric-maynard reviewed Jul 9, 2025

View reviewed changes

spec/polaris-management-service.yml Outdated Show resolved Hide resolved

snazy previously approved these changes Jul 9, 2025

View reviewed changes

polaris-core/src/main/java/org/apache/polaris/core/storage/aws/AwsStorageConfigurationInfo.java Outdated Show resolved Hide resolved

github-project-automation bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board Jul 9, 2025

dimas-b dismissed snazy’s stale review via b889c1e July 9, 2025 14:28

singhpk234 reviewed Jul 9, 2025

View reviewed changes

polaris-core/src/main/java/org/apache/polaris/core/storage/aws/AwsStorageConfigurationInfo.java Outdated Show resolved Hide resolved

singhpk234 previously approved these changes Jul 9, 2025

View reviewed changes

dimas-b dismissed singhpk234’s stale review via 780f36b July 9, 2025 14:42

dimas-b force-pushed the s3-path-style branch from b889c1e to 780f36b Compare July 9, 2025 14:42

eric-maynard requested changes Jul 10, 2025

View reviewed changes

github-project-automation bot moved this from Ready to merge to PRs In Progress in Basic Kanban Board Jul 10, 2025

snazy previously approved these changes Jul 10, 2025

View reviewed changes

dimas-b dismissed snazy’s stale review via 090b9b6 July 10, 2025 22:50

dimas-b force-pushed the s3-path-style branch from 780f36b to 090b9b6 Compare July 10, 2025 22:50

dimas-b requested review from eric-maynard, singhpk234 and snazy July 10, 2025 22:51

dimas-b force-pushed the s3-path-style branch from 090b9b6 to 09f5d77 Compare July 10, 2025 23:10

dimas-b force-pushed the s3-path-style branch 2 times, most recently from 6157bc0 to 85bbabe Compare July 15, 2025 16:27

dimas-b added 3 commits July 15, 2025 19:03

review: rename to getPathStyleAccess

2db692c

review: PATH_STYLE_ACCESS_DEFAULT, ternary

f21df95

dimas-b added 2 commits July 15, 2025 19:03

fix config round trip, add tests

0f14287

review: add Open API default for pathStyleAccess

47fc30e

dimas-b force-pushed the s3-path-style branch from 85bbabe to 47fc30e Compare July 15, 2025 23:03

eric-maynard previously approved these changes Jul 16, 2025

View reviewed changes

github-project-automation bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board Jul 16, 2025

eric-maynard reviewed Jul 16, 2025

View reviewed changes

review: remove default from doc comment for pathStyleAccess

21fe61c

dimas-b dismissed eric-maynard’s stale review via 21fe61c July 16, 2025 14:01

snazy approved these changes Jul 16, 2025

View reviewed changes

dimas-b merged commit 987c554 into apache:main Jul 16, 2025
12 checks passed

dimas-b deleted the s3-path-style branch July 16, 2025 15:46

github-project-automation bot moved this from Ready to merge to Done in Basic Kanban Board Jul 16, 2025

snazy added a commit to snazy/polaris that referenced this pull request Jul 17, 2025

Python client: add support for endpoint, sts-endpoint, path-style-access

449ae9a

Amends apache#1913 and apache#2012

snazy mentioned this pull request Jul 17, 2025

Python client: add support for endpoint, sts-endpoint, path-style-access #2127

Merged

snazy added a commit that referenced this pull request Jul 21, 2025

Python client: add support for endpoint, sts-endpoint, path-style-acc…

3316f4e

…ess (#2127) This change adds support for endpoint, sts-endpoint, path-style-access to the Polaris Python client. Amends #1913 and #2012

dimas-b mentioned this pull request Sep 25, 2025

Add KMS support for S3 #1424

Open

dimas-b mentioned this pull request Nov 20, 2025

adding support to use a kms key for s3 buckets data encryption (AWS only) #2802

Open

Add pathStyleAccess to AwsStorageConfigInfo #2012

Add pathStyleAccess to AwsStorageConfigInfo #2012

Uh oh!

Conversation

dimas-b commented Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

eric-maynard Jul 9, 2025

Choose a reason for hiding this comment

Uh oh!

dimas-b Jul 9, 2025

Choose a reason for hiding this comment

Uh oh!

eric-maynard Jul 9, 2025

Choose a reason for hiding this comment

Uh oh!

dimas-b Jul 9, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

singhpk234 left a comment

Choose a reason for hiding this comment

Uh oh!

dimas-b commented Jul 9, 2025

Uh oh!

eric-maynard left a comment

Choose a reason for hiding this comment

Uh oh!

snazy left a comment

Choose a reason for hiding this comment

Uh oh!

dimas-b commented Jul 14, 2025

Uh oh!

eric-maynard Jul 16, 2025

Choose a reason for hiding this comment

Uh oh!

dimas-b Jul 16, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Add `pathStyleAccess` to AwsStorageConfigInfo #2012

Add `pathStyleAccess` to AwsStorageConfigInfo #2012

dimas-b commented Jul 8, 2025 •

edited

Loading