Support S3 storage that does not have STS #2672

dimas-b · 2025-09-24T21:41:17Z

This change is backward compatible with old catalogs that have storage configuration for S3 systems with STS.

Add new property to S3 storage config: stsUnavailable (defaults to "available").
Do not call STS when unavailable in AwsCredentialsStorageIntegration, but still put other properties (e.g. s3.endpoint) into AccessConfig

Relates to #2615
Relates #2207

dennishuo · 2025-09-25T03:58:42Z

CHANGELOG.md


 - Added a Management API endpoint to reset principal credentials, controlled by the `ENABLE_CREDENTIAL_RESET` (default: true) feature flag.

+- Added support for S3-compatible storage that does not have STS (use `stsUavailable: true` in catalog storage configuration)


Could be worth expanding this sentence with a bit more of what's in the yaml for easy reference if people are reading the notes without wanting to read the full spec change -- specifically to clarify that it disables credential vending rather than vending some kind of root credentials after skipping STS subscoping.

How does the behavior compare to setting SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION=true?

SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION=true disables all storage config in API responses. Disabling STS only removes credentials.

Re: message changes: I'm not sure it's worth adding YAML to CHANGELOG... Would it not make it too verbose? In the end CHANGELOG has notes for many releases 🤔

How about I update it with a CLI example when I add CLI support for the new config entry?

I agree that from an user standpoint, CLI (or curl :) ) and updating the configuration list on the documentation would be more than welcome.

I will certainly add a docs page about this to "getting started" once the impl. is approved :)

@dennishuo : are you ok with this path forward? 🙂

Yeah, to clarify, I didn't mean to add too much, just maybe adding a ; credential vending will then be disabled in the parentheses so that the change note would say

(use `stsUavailable: true` in catalog storage configuration; credential vending will then be disabled);

In any case, non-blocking.

Incidentally, SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION really was only intended to pertain to the "credential" aspect, so it's unfortunate that it sort of fell into fully disabling StorageConfig mix-ins even for the non-credential-related config. But we can sort that out some other time.

singhpk234 · 2025-09-26T18:36:34Z

...core/src/main/java/org/apache/polaris/core/storage/aws/AwsCredentialsStorageIntegration.java

-                  String.valueOf(i.toEpochMilli()));
-            });
+
+    if (storageConfig.shouldUseSts()) {


[doubt] what if the client says it wan't to do x-Iceberg-vended-credentials, indication it wants credentials vending, but the storage doesn't support it, should we throw a 400 instead ?

Ideally yes, and that was in my first revision of this PR. Unfortunately many existing use cases excessively request credential vending for FILE storage and would be broken if we were to enforce this logic "in general".

We cannot enforce it here, because "storage integration" code has valid call paths without credential vending, where it exposes non-credential config to clients (e.g. S3 endpoints).

I'd like to enforce stricter checks as you suggested, but due to risk of breaking existing clients (I saw that in our reg. tests), I propose to defer it. WDYT?

I see, thanks for the explanation, I totally missed taking FILE into consideration ! IMHO its relatively easy to start with a stricter behaviour first and relax it later rather doing the other way as now there might be clients out there who would expect the behaviour to fail open and now expect 200, which we later plan to change to 400 anyways.

I would have appreciated to discussed this more !

I'll make a follow-up PR for this presently.

@singhpk234 : thanks for the ping and sorry about missing a related bug (should be fixed in #2711)

flyrain · 2025-09-26T18:49:51Z

polaris-core/src/main/java/org/apache/polaris/core/storage/aws/AwsStorageConfigurationInfo.java


+  /**
+   * Flag indicating whether STS is available or not. It is modeled in the negative to simplify
+   * support for unset values ({@code null} being interpreted as {@code false}).


Suggested change

* support for unset values ({@code null} being interpreted as {@code false}).

* support for unset values ({@code null} being interpreted as {@code true}).

Javadoc is correct, I believe 😅 A null (unset) value would be treated as StsUnavailable being false , meaning that STS is available (current default). Cf. shouldUseSts().

I believe interpreting null as false is quite common.

You are right, sorry for the misleading. I was confused by two new methods introduced, getStsUnavailable(), shouldUseSts(), I think it'd be nice to only keep one of them. For example, the check brought up by @singhpk234 here #2672 (comment), would be reasonable to apply when we check getStsUnavailable() purely. Wrapping getStsUnavailable() with shouldUseSts() makes it even more ambiguous, so that the logic become less obviously when we need to check if users ask for credential vending.

Ok, I removed shouldUseSts() from the config class. @flyrain : PTAL.

Thanks for doing that, this LGTM.

This change is backward compatible with old catalogs that have storage configuration for S3 systems with STS. * Add new property to S3 storage config: `stsUnavailable` (defaults to "available"). * Do not call STS when unavailable in `AwsCredentialsStorageIntegration`, but still put other properties (e.g. s3.endpoint) into `AccessConfig` Relates to apache#2615 Closes apache#2207

…s `true`

snazy

LGTM

The CLI should support the new option as well.

This is a follow-up change to apache#2672 and apache#2589 * Fix a big in `delegationModes` parameter propagation in `createTableStaged()` * Add checks (leading to 400) that credentials are vended when requested. * The check is disabled if `SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION` is set * Disable credential vending tests in file-based catalog since the "FILE" storage integration code never vends any credentials. These tests are still executed under `PolarisRestCatalogMinIOIT`

This is a follow-up change to apache#2672 * Add checks (leading to 400) that credentials are vended when requested. * The check is disabled if `SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION` or `ALLOW_EMPTY_VENDED_CREDENTIALS` are set * Disable credential vending tests in file-based catalog since the "FILE" storage integration code never vends any credentials. These tests are still executed under `PolarisRestCatalogMinIOIT` * Use `ALLOW_EMPTY_VENDED_CREDENTIALS=true` in `PolarisAuthzTestBase` since those tests make direct java calls to "load with access delegation" methods, but the test infra is not set up for credential vending. * Enable `ALLOW_EMPTY_VENDED_CREDENTIALS` in python client tests since PyIceberg requests credential vending by default, but the tests use FILE storage, for which Polaris does not vend credentials.

This is a follow-up change to apache#2672 striving to improve user-facing error reporting for S3 storage systems without STS. * Update call paths leading to `AccessConfig` to indicate whether the caller requires credentials to be vended or not. * Add checks to `AwsCredentialsStorageIntegration` (leading to 400) that S3 storage credentials are vended when requested. * Only those S3 systems where STS is not available (or disabled / not permitted) are affected. * Other storage integrations are not affected by this PR.

This is a follow-up change to apache#2672 striving to improve user-facing error reporting for S3 storage systems without STS. * Add property to `AccessConfig` to indicate whether the backing storage integration can produce credentials. * Add a check to `IcebergCatalogHandler` (leading to 400) that storage credentials are vended when requested and the backend is capable of vending credentials in principle. * Update `PolarisStorageIntegrationProviderImpl` to indicate that FILE storage does not support credential vending (requesitng redential vending with FILE storage does not produce any credentials and does not flag an error, which matches current Polaris behaviour). * Only those S3 systems where STS is not available (or disabled / not permitted) are affected. * Other storage integrations are not affected by this PR.

This is a follow-up change to #2672 striving to improve user-facing error reporting for S3 storage systems without STS. * Add property to `AccessConfig` to indicate whether the backing storage integration can produce credentials. * Add a check to `IcebergCatalogHandler` (leading to 400) that storage credentials are vended when requested and the backend is capable of vending credentials in principle. * Update `PolarisStorageIntegrationProviderImpl` to indicate that FILE storage does not support credential vending (requesitng redential vending with FILE storage does not produce any credentials and does not flag an error, which matches current Polaris behaviour). * Only those S3 systems where STS is not available (or disabled / not permitted) are affected. * Other storage integrations are not affected by this PR.

Following up on apache#2672, add new `--no-sts` option to CLI to allow configuring `stsUnavailable` in `AwsStorageConfigInfo`

* Add --no-sts to CLI Following up on #2672, add new `--no-sts` option to CLI to allow configuring `stsUnavailable` in `AwsStorageConfigInfo`

* (Based on PR#2223)Support Namespace/Table level RBAC for external passthrough catalogs (apache#2673) Creates missing synthetic entities for securables in external passthrough catalogs. Based on Option 1 discussed in the RBAC section of catalog federation design doc. In the future, we could remove calls to PolarisEntity.Builder() and replace them with entities fetched from the remote catalog. (enabling Option 2). --------- Co-authored-by: Pooja Nilangekar <poojan@umd.edu> * Docs: Add more details about v1 schema user to upgrade from 1.0 to 1.1 (apache#2674) * Site: The link https://iceberg.apache.org/concepts/catalog/ doesn't exist anymore. (apache#2683) * Docs: Add analytics for polaris.apache.org (apache#2676) * Make ENABLE_SUB_CATALOG_RBAC_FOR_FEDERATED_CATALOGS configurable per catalog (apache#2688) * Update ENABLE_SUB_CATALOG_RBAC_FOR_FEDERATED_CATALOGS to be configurable per catalog * chore(deps): update postgres docker tag to v18 (apache#2692) * fix(deps): update dependency org.eclipse.persistence:eclipselink to v4.0.8 (apache#2682) * fix(deps): update dependency org.apache.logging.log4j:log4j-core to v2.25.2 (apache#2646) * chore(deps): update dependency openapi-generator-cli to v7.15.0 (apache#2410) * chore(deps): update dependency io.quarkus to v3.27.0 (apache#2663) Co-authored-by: Mend Renovate <bot@renovateapp.com> * Publish Develocity builds scans for PRs and local use (apache#2596) This PR enables Develocity build scans for all PRs and contributors w/o an Apache account. CI build scans in the `apache/polaris` repo against branches and tags and having access to the ASF's Develocity secret continue to publish to the ASF's Develocity instance (no behavioral change). All other build scans are published to Gradle's public Develocity instance: - Build scans from local developer (non-CI) runs are only published, if Gradle is invoked with the `--scan` option. - Build scans from or targeting another repository than `apache/polaris` do need be enabled explicity by accepting Gradle's terms of service, via a repository variable, because this is a decision of the owner of a repository. Advanced options to configure another Develocity server or project-ID are available (for non-`apache/polaris` repositories). Detailed instructions in the `README.md`. * Fix & enhancements to the Events API hierarchy (apache#2629) Summary of changes: - Turned `PolarisEventListener` into an interface to facilitate implementation / mocking - Added missing `implements PolarisEvent` to many event records - Removed unused method overrides - Added missing method overrides to `TestPolarisEventListener` * fix(deps): update dependency org.kordamp.gradle:jandex-gradle-plugin to v2.3.0 (apache#2694) * Auth: reorganize internal authentication components (apache#2634) This PR contains no functional and no user-facing change. It is merely a refactor to better organize auth code. Summary of changes: - Moved all internal authentication components to the `org.apache.polaris.service.auth.internal` package and subpackages - Reduced visibility of utility classes - Renamed `TokenBroker` class hierarchy to stick to the naming standard: `<Algorithm>JWTBroker` - Introduced `@PolarisImmutable` whenever appropriate - Removed unused `NoneTokenBrokerFactory` (we already have `DisabledOAuth2ApiService`) - Removed unused `TokenBrokerFactoryConfig` * Enhancement : adding support for Aurora postgres AWS IAM authentication (apache#2650) Add support for postgres AWS IAM authentication using the `apache-client` lib. * Remove unused `name` arg from findCatalogByName in PolarisAdminService (apache#2691) * remove unused name param * Rename for better readability * Fix a race condition in sendNotification where concurrent parent-namespace creation causes failures (apache#2693) * Fix a race condition in sendNotification where concurrent parent-namespace creation causes failures The semantics of the createNonExistingNamespaces method used during sendNotification were supposed to be "create if needed". However, the behavior ended up surfacing an AlreadyExistsException if multiple concurrent sendNotification attempts were made for a brand-new namespace (where the notifications may be different tables). This would cause a table sync to fail if a sibling table was being synced at the same time, even though the new table should successfully get created under the shared namespace. * Also better future-proof the createNamespaceInternal logic by explicitly checking for ENTITY_ALREADY_EXISTS, per review suggestion. Log a less scary message since it's not an error scenario type of race condition, per review suggestion * Client: add credential reset option (apache#2698) * Client: add credential reset option * Client: add credential reset option * Client: add credential reset option * Add integration testing * Fix lint * fix(deps): update dependency software.amazon.awssdk:bom to v2.34.5 (apache#2702) * fix(deps): update dependency com.gradleup.shadow:shadow-gradle-plugin to v9.2.2 (apache#2661) * Support S3 storage that does not have STS (apache#2672) * Support S3 storage that does not have STS This change is backward compatible with old catalogs that have storage configuration for S3 systems with STS. * Add new property to S3 storage config: `stsUnavailable` (defaults to "available"). * Do not call STS when unavailable in `AwsCredentialsStorageIntegration`, but still put other properties (e.g. s3.endpoint) into `AccessConfig` Relates to apache#2615 Relates apache#2207 * Docs/improve idp documentation (apache#2695) * Fix Github links in IDP documentation * Separate IDP docs for usage and development * - Add telemetry config example - Fix link to getting started from landing page - Fix mentioning role-arn as required * Fix some relative links (local Hugo resolves them properly, but PR auto checks still fails) * Docs: narrow down --role-arn usage for AWS S3 only; fix a link in keycloak guide. * Docs: fix a link in keycloak guide. * chore(deps): update gradle/actions digest to 748248d (apache#2708) * Client: fix integration testing (apache#2700) * Add fallback in case the VERSION table is not present (apache#2653) * initial commit * wire up * pastefix * change to postgres specific code * [Catalog Federation] Add feature flag to disallow setting sub-RBAC for federated catalog at catalog level (apache#2696) In apache#2688 (comment), we've identified that configuring polaris.config.enable-sub-catalog-rbac-for-federated-catalogs at catalog level should not be allowed in all cases, especially when the owner is not the same subject as the catalog user or admin. This PR add a feature flag, ALLOW_SETTING_SUB_CATALOG_RBAC_FOR_FEDERATED_CATALOGS to allow owner to disable catalog level setting polaris.config.enable-sub-catalog-rbac-for-federated-catalogs * Fix `delegationModes` parameter propagation in `createTableStaged()` (apache#2713) This is follow-up bugfix for apache#2589 The bugfix part apache#2711 is extracted here since apache#2711 proved to be non-trivial and may require extra time. * Use the `delegationModes` method parameter as intended (as opposed to a local constant). * Generate Request IDs (if not specified); Return Request ID as a Header (apache#2602) * fix(deps): update dependency org.junit:junit-bom to v5.14.0 (apache#2715) * NoSQL persistence: add Java/Vert.X executor abstraction layer (apache#2527) Provides an abstraction to submit asynchronous tasks, optionally with a delay or delay + repetition and implementations based on Java's `ThreadPoolExecutor` and Vert.X. * Fix RDS devservices config + adopt for `:polaris-admin:test` (apache#2723) Changes: * Disables devservices for `:polaris-admin` tests as well, which is necessary to _not_ spin up test containers. * Use the explicit devservices-config as everywhere else. The first bullet point can cause excessive memory usage, especially with more test classes, eventually killing the whole GH runner. * fix(deps): update dependency io.smallrye:jandex to v3.5.0 (apache#2722) * fix(deps): update dependency org.jboss.weld:weld-junit5 to v5.0.2.final (apache#2721) * chore(deps): update quay.io/keycloak/keycloak docker tag to v26.4.0 (apache#2719) * Last merged commit 4024557 * NoSQL: Minor-ish changes to "nodes" projects Adopt nodes projects to OSS PR content * NoSQL: adapt to async package rename * Build: remove unnecessary explicit vertx-core dependency The async-vertx implementation should not propagate a different Vert.X dependency than Quarkus provides. This wouldn't be an issue if we could just use `enforcedPlatform()` for all Quarkus-builds, but sadly we cannot for the spark-plugin-inttests. --------- Co-authored-by: Honah (Jonas) J. <honahx@apache.org> Co-authored-by: Pooja Nilangekar <poojan@umd.edu> Co-authored-by: Prashant Singh <35593236+singhpk234@users.noreply.github.com> Co-authored-by: JB Onofré <jbonofre@apache.org> Co-authored-by: Mend Renovate <bot@renovateapp.com> Co-authored-by: Alexandre Dutra <adutra@apache.org> Co-authored-by: fabio-rizzo-01 <fabio.rizzocascio@jpmorgan.com> Co-authored-by: Dennis Huo <7410123+dennishuo@users.noreply.github.com> Co-authored-by: Yong Zheng <yongzheng0809@gmail.com> Co-authored-by: Dmitri Bourlatchkov <dmitri.bourlatchkov@gmail.com> Co-authored-by: olsoloviov <40199597+olsoloviov@users.noreply.github.com> Co-authored-by: Eric Maynard <eric.maynard+oss@snowflake.com> Co-authored-by: Adnan Hemani <adnan.h@berkeley.edu>

* Build: remove code to post-process generated Quarkus jars (apache#2667) Before Quarkus 3.28, the Quarkus generated jars used the "current" timestamp for all ZIP entries, which made the jars not-reproducible. Since Quarkus 3.28, the generated jars use a fixed timestamp for all ZIP entries, so the custom code is no longer necessary. This PR depends on Quarkus 3.28. * Update docker.io/jaegertracing/all-in-one Docker tag to v1.74.0 (apache#2751) * Updating metastore documentation with Aurora postgres example (apache#2706) * added Aurora postgres to metastore documentation * Service: Add events for APIs awaiting API changes (apache#2712) * fix(enhancement): add .idea, .vscode, .venv to top level .gitignore (apache#2718) fix(enhancement): add .idea, .vscode, .venv to top level .gitignore * Fix javadocs of `PolarisPrincipal.getPrincipalRoles()` (apache#2752) * fix(enhancement): squash commits (apache#2643) * fix(deps): update dependency io.smallrye.config:smallrye-config-core to v3.14.1 (apache#2755) * Extract interface for RequestIdGenerator (apache#2720) Summary of changes: 1. Extracted an interface from `RequestIdGenerator`. 2. The `generateRequestId` method now returns a `Uni<String>` in case custom implementations need to perform I/O or other blocking calls during request ID generation. 3. Also addressed comments in apache#2602. * JDBC: Handle schema evolution (apache#2714) * Deprecate legacy management endpoints for removal (apache#2749) * Deprecate LegacyManagementEndpoints for removal * Add PolarisResolutionManifestCatalogView.getResolvedCatalogEntity helper (apache#2750) this centralizes some common code and simplifies some test setups * Enforce that S3 credentials are vended when requested (apache#2711) This is a follow-up change to apache#2672 striving to improve user-facing error reporting for S3 storage systems without STS. * Add property to `AccessConfig` to indicate whether the backing storage integration can produce credentials. * Add a check to `IcebergCatalogHandler` (leading to 400) that storage credentials are vended when requested and the backend is capable of vending credentials in principle. * Update `PolarisStorageIntegrationProviderImpl` to indicate that FILE storage does not support credential vending (requesitng redential vending with FILE storage does not produce any credentials and does not flag an error, which matches current Polaris behaviour). * Only those S3 systems where STS is not available (or disabled / not permitted) are affected. * Other storage integrations are not affected by this PR. * [Catalog Federation] Ignore JIT entities when deleting federated catalogs, add integration test for namespace/table-level RBAC (apache#2690) When enabling table/namespace level RBAC in federated catalog, JIT entities will be created during privilege grant. In the short term, we should ignore them when dropping the catalog. In the long term, we will clean-up those entities when deleting the catalog. This will be the first step towards JIT entity clean-up: 1. Ignore JIT entities when dropping federated catalog (orphan entities) 2. Register tasks/in-place cleanup JIT entities during catalog drop 3. Add new functionality to PolarisMetastoreManager to support atomic delete non-used JIT entities during revoke. 4. Global Garbage Collector to clean-up unreachable entities (entities with non-existing catalog path/parent) * SigV4 Auth Support for Catalog Federation - Part 3: Service Identity Info Injection (apache#2523) This PR introduces service identity management for SigV4 Auth Support for Catalog Federation. Unlike user-supplied parameters, the service identity represents the identity of the Polaris service itself and should be managed by Polaris. * Service Identity Injection * Return injected service identity info in response * Use AwsCredentialsProvider to retrieve the credentials * Move some logic to ServiceIdentityConfiguration * Rename ServiceIdentityRegistry to ServiceIdentityProvider * Rename ResolvedServiceIdentity to ServiceIdentityCredential * Simplify the logic and add more tests * Use SecretReference and fix some small issues * Disable Catalog Federation * Update actions/stale digest to 5f858e3 (apache#2758) * Service: RealmContextFilter test refactor (apache#2747) * Update dependency software.amazon.awssdk:bom to v2.35.0 (apache#2760) * Update apache/spark Docker tag to v3.5.7 (apache#2727) * Update eric-maynard Team entry (apache#2763) I'm no longer affiliated with Snowflake, so we should update this page accordingly * Refactor resolutionManifest handling in PolarisAdminService (apache#2748) - remove mutable `resolutionManifest` field in favor of letting the "authorize" methods return their `PolarisResolutionManifest` - replace "find" helpers with "get" helpers that have built-in error handling * Implement Finer Grained Operations and Privileges For Update Table (apache#2697) This implements finer grained operations and privileges for update table in a backwards compatible way as discussed on the mailing list. The idea is that all the existing privileges and operations will work and continue to work even after this change. (i.e. TABLE_WRITE_PROPERTIES will still ensure update table is authorized even after these changes). However, because Polaris will now be able to identify each operation within an UpdateTable request and has a privilege model with inheritance that maps to each operation, users will now have the option of restricting permissions at a finer level if desired. * [Python CLI][CI Failure] Pin pydantic version to < 2.12.0 to fix CI failure (apache#2770) * Delete ServiceSecretReference (apache#2768) * JDBC: Fix Bootstrap with schema options (apache#2762) * Site: Add puppygraph integration (apache#2753) * Update Changelog with finer grained authz (apache#2775) * Add Arguments to Various Event Records (apache#2765) * Update immutables to v2.11.5 (apache#2776) * Client: add support for policy management (apache#2701) Implementation for policy management via Polaris CLI (apache#1867). Here are the subcommands to API mapping: attach - PUT /polaris/v1/{prefix}/namespaces/{namespace}/policies/{policy-name}/mappings create - POST /polaris/v1/{prefix}/namespaces/{namespace}/policies/{policy-name}/mappings delete - DELETE /polaris/v1/{prefix}/namespaces/{namespace}/policies/{policy-name} detach - POST /polaris/v1/{prefix}/namespaces/{namespace}/policies/{policy-name}/mappings get - GET /polaris/v1/{prefix}/namespaces/{namespace}/policies/{policy-name} list - GET /polaris/v1/{prefix}/namespaces/{namespace}/policies - This is default for `list` operation - GET /polaris/v1/{prefix}/applicable-policies - This is when we have `--applicable` option provided update - PUT /polaris/v1/{prefix}/namespaces/{namespace}/policies/{policy-name} * Update dependency com.google.cloud:google-cloud-storage-bom to v2.58.1 (apache#2764) * Update dependency org.jboss.weld:weld-junit5 to v5.0.3.Final (apache#2777) * Update the LICENSE and NOTICE files in the runtime (apache#2779) * SigV4 Auth Support for Catalog Federation - Part 4: Connection Credential Manager (apache#2759) This PR introduces a flexible credential management system for Polaris. Building on Part 3's service identity management, this system combines Polaris service identities with user-provided authentication parameters to generate credentials for remote catalog access. The core of this PR is the new ConnectionCredentialVendor interface, which: Generates connection credentials by combining service identity with user auth parameters Supports different authentication types (AWS SIGV4, AZURE Entra, GCP IAM) through CDI, currently only supports SigV4. Provides on-demand credential generation Enables easy extension for new authentication types In the long term, we should move the storage credential management logic out of PolarisMetastoreManager, PolarisMetastoreManager should only provide persistence interfaces. * Extract IcebergCatalog.getAccessConfig to a separate class AccessConfigProvider (apache#2736) This PR extracts credential vending entrypoint getAccessConfig from IcebergCatalog into a new centralized AccessConfigProvider class, decoupling credential generation from catalog implementations. The old SupportsCredentialVending is removed in this PR upon discussion * Update immutables to v2.11.6 (apache#2780) * Enhance Release docs (apache#2787) * Spark: Remove unnecessary dependency (apache#2789) * Update Pull Request Template (apache#2788) * Freeze 1.2 change log (apache#2783) * [Catalog Federation] Enable Credential Vending for Passthrough Facade Catalog (apache#2784) This PR introduces credential vending support for passthrough-facade catalogs. When creating a passthrough-facade catalog, the configuration currently requires two components: StorageConfig – specifies the storage info for the remote catalog. ConnectionInfo – defines connection parameters for the underlying remote catalog. With this change, the StorageConfig is now also used to vend temporary credentials for user requests. Credential vending honors table-level RBAC policies to determine whether to issue read-only or read-write credentials, ensuring access control consistency with Polaris authorization semantics. A new test case validates the credential vending workflow, verifying both read and write credential vending. Note: the remote catalog referenced by the passthrough-facade does not need to support IRC * Site: Add docs for catalog federation (apache#2761) * Python client: update CHANGELOG.MD for recent changes (apache#2796) * Python client: remove Python 3.9 support (apache#2795) * Update dependency software.amazon.awssdk:bom to v2.35.5 (apache#2799) * FIX REG tests with cloud providers (apache#2793) * [Catalog Federation] Block credential vending for remote tables outside allowed location list (apache#2791) * Correct invalid example in management service OpenAPI spec (apache#2801) The `example` was incorrectly placed as a sibling of `$ref` within a `schema` object in `polaris-management-service.yml`. According to the OpenAPI specification, properties that are siblings of a `$ref` are ignored. This was causing a `NullPointerException` in OpenAPI Generator v7.13.0+ due to a change in how examples are processed. The generator now expects all `examples` to be valid and non-empty, and a misplaced `example` can lead to a null reference when the generator tries to access it (we are not yet using v7.13.0+, thus not a problem at the moment). This commit moves the `example` to be a sibling of the `schema` object, which is the correct placement according to the OpenAPI specification. Reference error when using newer version of openapi-generator-cli: ``` openapi-generator-cli generate -i spec/polaris-catalog-service.yaml -g python -o client/python --additional-properties=packageName=polaris.catalog --additional-properties=apiNameSuffix="" --skip-validate-spec --additional-properties=pythonVersion=3.13 --ignore-file-override /local/client/python/.openapi-generator-ignore ... Exception: Cannot invoke "io.swagger.v3.oas.models.examples.Example.getValue()" because the return value of "java.util.Map.get(Object)" is null at org.openapitools.codegen.DefaultGenerator.processOperation(DefaultGenerator.java:1606) at org.openapitools.codegen.DefaultGenerator.processPaths(DefaultGenerator.java:1474) at org.openapitools.codegen.DefaultGenerator.generateApis(DefaultGenerator.java:663) at org.openapitools.codegen.DefaultGenerator.generate(DefaultGenerator.java:1296) at org.openapitools.codegen.cmd.Generate.execute(Generate.java:535) at org.openapitools.codegen.cmd.OpenApiGeneratorCommand.run(OpenApiGeneratorCommand.java:32) at org.openapitools.codegen.OpenAPIGenerator.main(OpenAPIGenerator.java:66) Caused by: java.lang.NullPointerException: Cannot invoke "io.swagger.v3.oas.models.examples.Example.getValue()" because the return value of "java.util.Map.get(Object)" is null at org.openapitools.codegen.utils.ExamplesUtils.unaliasExamples(ExamplesUtils.java:75) at org.openapitools.codegen.DefaultCodegen.unaliasExamples(DefaultCodegen.java:2343) at org.openapitools.codegen.DefaultCodegen.fromResponse(DefaultCodegen.java:4934) at org.openapitools.codegen.DefaultCodegen.fromOperation(DefaultCodegen.java:4575) at org.openapitools.codegen.DefaultGenerator.processOperation(DefaultGenerator.java:1574) ... 6 more ``` * Update dependency io.opentelemetry:opentelemetry-bom to v1.55.0 (apache#2804) * Update dependency io.micrometer:micrometer-bom to v1.15.5 (apache#2806) * Bump version for python deps (apache#2800) * bump version for python deps * bump version for python deps * bump version for python deps * Update openapi-generatr-cli from 7.11.0.post0 to 7.12.0 * Pin poetry version * Pin poetry version * Update dependency io.projectreactor.netty:reactor-netty-http to v1.2.11 (apache#2809) * [Catalog Federation] Add Connection Credential Vendors for Other Auth Types (apache#2782) Add Connection Credential Vendors for Other Auth Types This change is a prerequisite for enabling connection credential caching. By making PolarisCredentialManager the central entry point for obtaining connection credentials, we can introduce caching cleanly and manage all credential flows in a consistent way. * Last merged commit 6b957ec --------- Co-authored-by: Mend Renovate <bot@renovateapp.com> Co-authored-by: fabio-rizzo-01 <fabio.rizzocascio@jpmorgan.com> Co-authored-by: Adnan Hemani <adnan.h@berkeley.edu> Co-authored-by: Artur Rakhmatulin <artur.rakhmatulin@gmail.com> Co-authored-by: Alexandre Dutra <adutra@apache.org> Co-authored-by: Prashant Singh <35593236+singhpk234@users.noreply.github.com> Co-authored-by: Christopher Lambert <xn137@gmx.de> Co-authored-by: Dmitri Bourlatchkov <dmitri.bourlatchkov@gmail.com> Co-authored-by: Honah (Jonas) J. <honahx@apache.org> Co-authored-by: Rulin Xing <xjdkcsq3@gmail.com> Co-authored-by: Eric Maynard <eric.maynard+oss@snowflake.com> Co-authored-by: Travis Bowen <122238243+travis-bowen@users.noreply.github.com> Co-authored-by: Jaz Ku <jsku@dons.usfca.edu> Co-authored-by: Yong Zheng <yongzheng0809@gmail.com> Co-authored-by: JB Onofré <jbonofre@apache.org> Co-authored-by: Yufei Gu <yufei@apache.org>

* Update Quarkus Platform and Group to v3.28.4 (apache#2786) * Update dependency org.testcontainers:testcontainers-bom to v2.0.1 (apache#2830) * Build/polaris-core: Remove outdated `constraint`s (apache#2818) The `:polaris-core` build scripts contains (soft) version-constraints for some dependencies with a vague reason "Vulnerability detected in ..." (concrete CVE/reason not mentioned) referencing specific dependency versions. The mentioned versions are all quite outdated, some are even not transitively referenced. Hence, removing those constraings, as those seem no longer relevant. Effective dependency versions can be inspected via `./gradlew :polaris-core:dependencies --configuration runtimeClasspath`. * Add Community Meetings for 2025-10-02 and 2025-10-16 (apache#2832) * Update docker.io/prom/prometheus Docker tag to v3.7.1 (apache#2834) * testcontainers v2: tackle deprecation warnings (apache#2835) * Add findPrincipalById helper (apache#2810) * Add findPrincipalById helper this simplifies frequent usage of the lower level `loadEntity` api (similar to the existing `findPrincipalByName` helper) * [Python] Add more tests cases for policy CLI (apache#2831) * Update dependency software.amazon.awssdk:bom to v2.35.10 (apache#2840) * Update dependency ch.qos.logback:logback-classic to v1.5.20 (apache#2839) * Reproducible builds: make parent pom content reproducible (apache#2826) The parent pom contains the `<developer>` and `<contributor>` elements. The former is populated from ASF people information including role information (champion, mentor, chair, (P)PMC member, committer). The latter is retrieved from a GitHub API endpoint, ordered by number contributions. Especially the latter list is prone to vary between builds, which makes the parent pom not reproducible as the locally built one is likely different from the one that was built by the release managed (staged artifact). This change removes both lists, leaving a single static `<developer>` entry pointing to `https://polaris.apache.org/community/`. Related build-script code has been updated and no longer retrieves people information. * Log root cause exceptions in mappers (apache#2837) Fix `IcebergExceptionMapper` and `PolarisExceptionMapper` to pass exceptions as "cause" to the logger (as opposed to unreferenced log parameters). * Remove credential flag from `StorageAccessProperty.CLIENT_REGION` (apache#2838) `CLIENT_REGION` is not a credential value, which is in line with Iceberg's `VendedCredentialsProvider` code. Cf. apache/iceberg#11389 * CI: Let all workflows use GitHub's docker.io mirror (apache#2841) * Correct template rendering for authentication options (apache#2808) * Correct template rendering for authentication options * Added tpl back * Increase javadoc visibility in `:polaris-async-vertx` (apache#2745) This is to fix javadoc error: `No public or protected classes found to document` * Update slack invite url (apache#2846) * Remove unused ConcurrentLinkedQueueWithApproximateSize (apache#2849) * Merge AwsCloudWatchConfiguration and QuarkusAwsCloudWatchConfiguration (apache#2848) For some reason, these two classes weren't properly merged when the runtime-service and service-common modules were merged. This PR fixes that. This PR also adds some examples of AWS Cloud Watch configuration to the default application.properties file. * Move TestPolarisEventListener to test fixtures (apache#2850) * Update dependency com.google.cloud:google-cloud-storage-bom to v2.59.0 (apache#2857) * Update actions/stale digest to e46bbab (apache#2856) * Servcie: Remove a duplicated config (apache#2854) * Update docker.io/prom/prometheus Docker tag to v3.7.2 (apache#2858) * Update Quarkus Platform and Group to v3.28.5 (apache#2859) * Update dependency com.google.errorprone:error_prone_core to v2.43.0 (apache#2860) * Add --no-sts to CLI (apache#2855) * Add --no-sts to CLI Following up on apache#2672, add new `--no-sts` option to CLI to allow configuring `stsUnavailable` in `AwsStorageConfigInfo` * Use AccessConfigProvider.getAccessConfig in DefaultFileIOFactory (apache#2852) * CLI: Remove the trailing comma (apache#2863) * Update dependency pip-licenses-cli to v3 (apache#2842) * Update dependency pip-licenses-cli to v3 * Update pip-licenses-cli version format * Fix pip-licenses-cli version specification --------- Co-authored-by: Yong Zheng <yongzheng0809@gmail.com> * Update quay.io/keycloak/keycloak Docker tag to v26.4.2 (apache#2868) * Bump main to 1.3.0-SNAPSHOT (apache#2870) * Add properties from TableMetadata into Table entity internalProperties (apache#2735) * Add properties from TableMetadata into Table entity internalProperties * Made table properties constants and pulled out static utility method * Update dependency io.smallrye:jandex to v3.5.1 (apache#2872) * Fix exec flags on getting-started scripts (apache#2878) * Add `+x` to script source files * Remove (unnecessary) `chmod` from docs * Update plugin jcstress to v0.9.0 (apache#2882) * Update registry.access.redhat.com/ubi9/openjdk-21-runtime Docker tag to v1.23-6.1761164966 (apache#2874) * Update dependency openapi-generator-cli to v7.16.0 (apache#2703) * Update Gradle to v9 (apache#2226) * Update Gradle to v9 * adopt gradlew --------- Co-authored-by: Robert Stupp <snazy@snazy.de> * Last merged commit 7892540 --------- Co-authored-by: Mend Renovate <bot@renovateapp.com> Co-authored-by: JB Onofré <jbonofre@apache.org> Co-authored-by: Christopher Lambert <xn137@gmx.de> Co-authored-by: Nuoya Jiang <98131931+NuoyaJiang@users.noreply.github.com> Co-authored-by: Dmitri Bourlatchkov <dmitri.bourlatchkov@gmail.com> Co-authored-by: Yong Zheng <yongzheng0809@gmail.com> Co-authored-by: Honah (Jonas) J. <honahx@apache.org> Co-authored-by: Alexandre Dutra <adutra@apache.org> Co-authored-by: Yufei Gu <yufei@apache.org> Co-authored-by: Nuoya Jiang <98131931+CodingBangboo@users.noreply.github.com> Co-authored-by: Michael Collado <40346148+collado-mike@users.noreply.github.com>

github-project-automation bot added this to Basic Kanban Board Sep 24, 2025

github-project-automation bot moved this to PRs In Progress in Basic Kanban Board Sep 24, 2025

dimas-b force-pushed the non-sts-s3-catalog branch from e912210 to 8c393cb Compare September 24, 2025 21:42

dimas-b mentioned this pull request Sep 24, 2025

[ENHANCEMENT] Support non-AWS S3 storage that does not have STS #2207

Closed

dimas-b force-pushed the non-sts-s3-catalog branch from 8c393cb to a28accd Compare September 25, 2025 00:00

dennishuo reviewed Sep 25, 2025

View reviewed changes

dimas-b force-pushed the non-sts-s3-catalog branch from a28accd to 4b7c9e2 Compare September 25, 2025 15:22

dimas-b requested a review from dennishuo September 25, 2025 15:22

dimas-b added this to the 1.2.0 milestone Sep 26, 2025

singhpk234 reviewed Sep 26, 2025

View reviewed changes

flyrain reviewed Sep 26, 2025

View reviewed changes

dimas-b added 3 commits September 26, 2025 18:14

Allow setting stsUnavailable only if ALLOW_SETTING_S3_ENDPOINTS i…

69603b8

…s `true`

review: inline shouldUseSts()

60746be

dimas-b force-pushed the non-sts-s3-catalog branch from 4b7c9e2 to 60746be Compare September 26, 2025 22:14

dennishuo approved these changes Sep 28, 2025

View reviewed changes

github-project-automation bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board Sep 28, 2025

snazy approved these changes Sep 29, 2025

View reviewed changes

dimas-b merged commit f97c5eb into apache:main Sep 29, 2025
14 checks passed

github-project-automation bot moved this from Ready to merge to Done in Basic Kanban Board Sep 29, 2025

dimas-b deleted the non-sts-s3-catalog branch September 29, 2025 14:11

dimas-b mentioned this pull request Sep 29, 2025

Enforce that S3 credentials are vended when requested #2711

Merged

rohangoli mentioned this pull request Oct 1, 2025

Support Dell ECS On-Premise S3 Object Store with STS support #2743

Open

dimas-b added a commit to dimas-b/polaris that referenced this pull request Oct 21, 2025

Add --no-sts to CLI

3e2c645

Following up on apache#2672, add new `--no-sts` option to CLI to allow configuring `stsUnavailable` in `AwsStorageConfigInfo`

dimas-b mentioned this pull request Oct 21, 2025

Add --no-sts to CLI #2855

Merged

dimas-b added a commit to dimas-b/polaris that referenced this pull request Oct 22, 2025

Add --no-sts to CLI

c7349d5

Following up on apache#2672, add new `--no-sts` option to CLI to allow configuring `stsUnavailable` in `AwsStorageConfigInfo`

dimas-b added a commit that referenced this pull request Oct 22, 2025

Add --no-sts to CLI (#2855)

1e356ea

* Add --no-sts to CLI Following up on #2672, add new `--no-sts` option to CLI to allow configuring `stsUnavailable` in `AwsStorageConfigInfo`


		- Added a Management API endpoint to reset principal credentials, controlled by the `ENABLE_CREDENTIAL_RESET` (default: true) feature flag.

		- Added support for S3-compatible storage that does not have STS (use `stsUavailable: true` in catalog storage configuration)

	* support for unset values ({@code null} being interpreted as {@code false}).
	* support for unset values ({@code null} being interpreted as {@code true}).

Support S3 storage that does not have STS #2672

Support S3 storage that does not have STS #2672

Uh oh!

Conversation

dimas-b commented Sep 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dimas-b Sep 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

snazy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

dimas-b commented Sep 24, 2025 •

edited

Loading

dimas-b Sep 26, 2025 •

edited

Loading