Dremio merge 2025 09 30 15 57 (apache#136)

* (Based on PR#2223)Support Namespace/Table level RBAC for external passthrough catalogs (apache#2673)

Creates missing synthetic entities for securables in external passthrough catalogs.
Based on Option 1 discussed in the RBAC section of catalog federation design doc.

In the future, we could remove calls to PolarisEntity.Builder() and replace them with entities fetched from the remote catalog. (enabling Option 2).

---------

Co-authored-by: Pooja Nilangekar <poojan@umd.edu>

* Docs: Add more details about v1 schema user to upgrade from 1.0 to 1.1 (apache#2674)

* Site: The link https://iceberg.apache.org/concepts/catalog/ doesn't exist anymore. (apache#2683)

* Docs: Add analytics for polaris.apache.org (apache#2676)

* Make ENABLE_SUB_CATALOG_RBAC_FOR_FEDERATED_CATALOGS configurable per catalog (apache#2688)

* Update ENABLE_SUB_CATALOG_RBAC_FOR_FEDERATED_CATALOGS to be configurable per catalog

* chore(deps): update postgres docker tag to v18 (apache#2692)

* fix(deps): update dependency org.eclipse.persistence:eclipselink to v4.0.8 (apache#2682)

* fix(deps): update dependency org.apache.logging.log4j:log4j-core to v2.25.2 (apache#2646)

* chore(deps): update dependency openapi-generator-cli to v7.15.0 (apache#2410)

* chore(deps): update dependency io.quarkus to v3.27.0 (apache#2663)

Co-authored-by: Mend Renovate <bot@renovateapp.com>

* Publish Develocity builds scans for PRs and local use (apache#2596)

This PR enables Develocity build scans for all PRs and contributors w/o an Apache account.

CI build scans in the `apache/polaris` repo against branches and tags and having access to the ASF's Develocity secret continue to publish to the ASF's Develocity instance (no behavioral change).

All other build scans are published to Gradle's public Develocity instance:
- Build scans from local developer (non-CI) runs are only published, if Gradle is invoked with the `--scan` option.
- Build scans from or targeting another repository than `apache/polaris` do need be enabled explicity by accepting Gradle's terms of service, via a repository variable, because this is a decision of the owner of a repository.

Advanced options to configure another Develocity server or project-ID are available (for non-`apache/polaris` repositories).

Detailed instructions in the `README.md`.

* Fix & enhancements to the Events API hierarchy (apache#2629)

Summary of changes:

- Turned `PolarisEventListener` into an interface to facilitate implementation / mocking
- Added missing `implements PolarisEvent` to many event records
- Removed unused method overrides
- Added missing method overrides to `TestPolarisEventListener`

* fix(deps): update dependency org.kordamp.gradle:jandex-gradle-plugin to v2.3.0 (apache#2694)

* Auth: reorganize internal authentication components (apache#2634)

This PR contains no functional and no user-facing change. It is merely a refactor to better organize auth code.

Summary of changes:

- Moved all internal authentication components to the `org.apache.polaris.service.auth.internal` package and subpackages
- Reduced visibility of utility classes
- Renamed `TokenBroker` class hierarchy to stick to the naming standard: `<Algorithm>JWTBroker`
- Introduced `@PolarisImmutable` whenever appropriate
- Removed unused `NoneTokenBrokerFactory` (we already have `DisabledOAuth2ApiService`)
- Removed unused `TokenBrokerFactoryConfig`

* Enhancement : adding support for Aurora postgres AWS IAM authentication (apache#2650)

Add support for postgres AWS IAM authentication using the `apache-client` lib.

* Remove unused `name` arg from findCatalogByName in PolarisAdminService (apache#2691)

* remove unused name param

* Rename for better readability

* Fix a race condition in sendNotification where concurrent parent-namespace creation causes failures (apache#2693)

* Fix a race condition in sendNotification where concurrent parent-namespace creation causes failures

The semantics of the createNonExistingNamespaces method used during sendNotification were supposed
to be "create if needed". However, the behavior ended up surfacing an AlreadyExistsException
if multiple concurrent sendNotification attempts were made for a brand-new namespace (where
the notifications may be different tables). This would cause a table sync to fail if a sibling
table was being synced at the same time, even though the new table should successfully get created
under the shared namespace.

* Also better future-proof the createNamespaceInternal logic by explicitly
checking for ENTITY_ALREADY_EXISTS, per review suggestion.

Log a less scary message since it's not an error scenario type of race
condition, per review suggestion

* Client: add credential reset option (apache#2698)

* Client: add credential reset option

* Client: add credential reset option

* Client: add credential reset option

* Add integration testing

* Fix lint

* fix(deps): update dependency software.amazon.awssdk:bom to v2.34.5 (apache#2702)

* fix(deps): update dependency com.gradleup.shadow:shadow-gradle-plugin to v9.2.2 (apache#2661)

* Support S3 storage that does not have STS (apache#2672)

* Support S3 storage that does not have STS

This change is backward compatible with old catalogs that have storage configuration for S3 systems with STS.

* Add new property to S3 storage config: `stsUnavailable` (defaults to "available").

* Do not call STS when unavailable in `AwsCredentialsStorageIntegration`, but still put other properties (e.g. s3.endpoint) into `AccessConfig`

Relates to apache#2615
Relates apache#2207

* Docs/improve idp documentation (apache#2695)

* Fix Github links in IDP documentation

* Separate IDP docs for usage and development

* - Add telemetry config example
- Fix link to getting started from landing page
- Fix mentioning role-arn as required

* Fix some relative links (local Hugo resolves them properly, but PR auto checks still fails)

* Docs: narrow down --role-arn usage for AWS S3 only; fix a link in keycloak guide.

* Docs: fix a link in keycloak guide.

* chore(deps): update gradle/actions digest to 748248d (apache#2708)

* Client: fix integration testing (apache#2700)

* Add fallback in case the VERSION table is not present (apache#2653)

* initial commit

* wire up

* pastefix

* change to postgres specific code

* [Catalog Federation] Add feature flag to disallow setting sub-RBAC for federated catalog at catalog level (apache#2696)

In apache#2688 (comment), we've identified that configuring polaris.config.enable-sub-catalog-rbac-for-federated-catalogs at catalog level should not be allowed in all cases, especially when the owner is not the same subject as the catalog user or admin.

This PR add a feature flag, ALLOW_SETTING_SUB_CATALOG_RBAC_FOR_FEDERATED_CATALOGS to allow owner to disable catalog level setting polaris.config.enable-sub-catalog-rbac-for-federated-catalogs

* Fix `delegationModes` parameter propagation in `createTableStaged()` (apache#2713)

This is follow-up bugfix for apache#2589

The bugfix part apache#2711 is extracted here since apache#2711 proved to be
non-trivial and may require extra time.

* Use the `delegationModes` method parameter as intended (as opposed
  to a local constant).

* Generate Request IDs (if not specified); Return Request ID as a Header (apache#2602)

* fix(deps): update dependency org.junit:junit-bom to v5.14.0 (apache#2715)

* NoSQL persistence: add Java/Vert.X executor abstraction layer (apache#2527)

Provides an abstraction to submit asynchronous tasks, optionally with a delay or delay + repetition and implementations based on Java's `ThreadPoolExecutor` and Vert.X.

* Fix RDS devservices config + adopt for `:polaris-admin:test` (apache#2723)

Changes:
* Disables devservices for `:polaris-admin` tests as well, which is necessary to _not_ spin up test containers.
* Use the explicit devservices-config as everywhere else.

The first bullet point can cause excessive memory usage, especially with more test classes, eventually killing the whole GH runner.

* fix(deps): update dependency io.smallrye:jandex to v3.5.0 (apache#2722)

* fix(deps): update dependency org.jboss.weld:weld-junit5 to v5.0.2.final (apache#2721)

* chore(deps): update quay.io/keycloak/keycloak docker tag to v26.4.0 (apache#2719)

* Last merged commit 4024557

* NoSQL: Minor-ish changes to "nodes" projects

Adopt nodes projects to OSS PR content

* NoSQL: adapt to async package rename

* Build: remove unnecessary explicit vertx-core dependency

The async-vertx implementation should not propagate a different Vert.X dependency than Quarkus provides. This wouldn't be an issue if we could just use `enforcedPlatform()` for all Quarkus-builds, but sadly we cannot for the spark-plugin-inttests.

---------

Co-authored-by: Honah (Jonas) J. <honahx@apache.org>
Co-authored-by: Pooja Nilangekar <poojan@umd.edu>
Co-authored-by: Prashant Singh <35593236+singhpk234@users.noreply.github.com>
Co-authored-by: JB Onofré <jbonofre@apache.org>
Co-authored-by: Mend Renovate <bot@renovateapp.com>
Co-authored-by: Alexandre Dutra <adutra@apache.org>
Co-authored-by: fabio-rizzo-01 <fabio.rizzocascio@jpmorgan.com>
Co-authored-by: Dennis Huo <7410123+dennishuo@users.noreply.github.com>
Co-authored-by: Yong Zheng <yongzheng0809@gmail.com>
Co-authored-by: Dmitri Bourlatchkov <dmitri.bourlatchkov@gmail.com>
Co-authored-by: olsoloviov <40199597+olsoloviov@users.noreply.github.com>
Co-authored-by: Eric Maynard <eric.maynard+oss@snowflake.com>
Co-authored-by: Adnan Hemani <adnan.h@berkeley.edu>

Author: 14 people

.github/actions/ci-incr-build-cache-prepare/action.yml

@@ -66,7 +66,7 @@ runs:
         fi
 
     - name: Gradle / Setup
-      uses: gradle/actions/setup-gradle@ed408507eac070d1f99cc633dbcf757c94c7933a # v4
+      uses: gradle/actions/setup-gradle@748248ddd2a24f49513d8f472f81c3a07d4d50e1 # v4
       with:
         cache-read-only: ${{ inputs.cache-read-only }}
         validate-wrappers: false

.github/workflows/gradle.yml

@@ -32,6 +32,11 @@ on:
   pull_request:
     branches: [ "main" ]
 
+env:
+  GRADLE_TOS_ACCEPTED: ${{ vars.GRADLE_TOS_ACCEPTED }}
+  DEVELOCITY_SERVER: ${{ vars.DEVELOCITY_SERVER }}
+  DEVELOCITY_PROJECT_ID: ${{ vars.DEVELOCITY_PROJECT_ID }}
+
 jobs:
 
   unit-tests:

.github/workflows/helm.yml

@@ -30,6 +30,11 @@ on:
   pull_request:
     branches: [ "main" ]
 
+env:
+  GRADLE_TOS_ACCEPTED: ${{ vars.GRADLE_TOS_ACCEPTED }}
+  DEVELOCITY_SERVER: ${{ vars.DEVELOCITY_SERVER }}
+  DEVELOCITY_PROJECT_ID: ${{ vars.DEVELOCITY_PROJECT_ID }}
+
 jobs:
 
   helm-tests:

.github/workflows/nightly.yml

@@ -40,6 +40,11 @@ permissions:
   security-events: read
   statuses: read
 
+env:
+  GRADLE_TOS_ACCEPTED: ${{ vars.GRADLE_TOS_ACCEPTED }}
+  DEVELOCITY_SERVER: ${{ vars.DEVELOCITY_SERVER }}
+  DEVELOCITY_PROJECT_ID: ${{ vars.DEVELOCITY_PROJECT_ID }}
+
 jobs:
   nightly_build:
     runs-on: ubuntu-latest

.github/workflows/python-client.yml

@@ -32,6 +32,11 @@ on:
   pull_request:
     branches: [ "main" ]
 
+env:
+  GRADLE_TOS_ACCEPTED: ${{ vars.GRADLE_TOS_ACCEPTED }}
+  DEVELOCITY_SERVER: ${{ vars.DEVELOCITY_SERVER }}
+  DEVELOCITY_PROJECT_ID: ${{ vars.DEVELOCITY_PROJECT_ID }}
+
 jobs:
   build:
 
@@ -67,15 +72,6 @@ jobs:
         run: |
           make client-unit-test
 
-      - name: Image build
-        env:
-          DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}
-        run: |
-          ./gradlew \
-            :polaris-server:assemble \
-            :polaris-server:quarkusAppPartsBuild --rerun \
-            -Dquarkus.container-image.build=true
-
       - name: Integration Tests
         run: |
           make client-integration-test

.github/workflows/regtest.yml

@@ -24,6 +24,11 @@ on:
   pull_request:
     branches: [ "main" ]
 
+env:
+  GRADLE_TOS_ACCEPTED: ${{ vars.GRADLE_TOS_ACCEPTED }}
+  DEVELOCITY_SERVER: ${{ vars.DEVELOCITY_SERVER }}
+  DEVELOCITY_PROJECT_ID: ${{ vars.DEVELOCITY_PROJECT_ID }}
+
 jobs:
   regtest:
 

.github/workflows/spark_client_regtests.yml

@@ -24,6 +24,11 @@ on:
   pull_request:
     branches: [ "main" ]
 
+env:
+  GRADLE_TOS_ACCEPTED: ${{ vars.GRADLE_TOS_ACCEPTED }}
+  DEVELOCITY_SERVER: ${{ vars.DEVELOCITY_SERVER }}
+  DEVELOCITY_PROJECT_ID: ${{ vars.DEVELOCITY_PROJECT_ID }}
+
 jobs:
   spark-plugin-regtest:
 

CHANGELOG.md

@@ -31,6 +31,7 @@ request adding CHANGELOG notes for breaking (!) changes and possibly other secti
 
 ### Upgrade Notes
 
+- Amazon RDS plugin enabled, this allows polaris to connect to AWS Aurora PostgreSQL using IAM authentication.
 - The EclipseLink Persistence implementation has been deprecated since 1.0.0 and will be completely removed
   in 1.3.0 or in 2.0.0 (whichever happens earlier).
 
@@ -39,6 +40,11 @@ request adding CHANGELOG notes for breaking (!) changes and possibly other secti
 ### New Features
 
 - Added a Management API endpoint to reset principal credentials, controlled by the `ENABLE_CREDENTIAL_RESET` (default: true) feature flag.
+- The `ENABLE_SUB_CATALOG_RBAC_FOR_FEDERATED_CATALOGS` was added to support sub-catalog (initially namespace and table) RBAC for federated catalogs.
+  The setting can be configured on a per-catalog basis by setting the catalog property: `polaris.config.enable-sub-catalog-rbac-for-federated-catalogs`.
+  The realm-level feature flag `ALLOW_SETTING_SUB_CATALOG_RBAC_FOR_FEDERATED_CATALOGS` (default: true) controls whether this functionality can be enabled or modified at the catalog level.
+
+- Added support for S3-compatible storage that does not have STS (use `stsUavailable: true` in catalog storage configuration)
 
 ### Changes
 
@@ -107,7 +113,10 @@ Apache Polaris 1.1.0-incubating was released on September 19th, 2025.
       ON CONFLICT (version_key) DO UPDATE
                               SET version_value = EXCLUDED.version_value;
       COMMENT ON TABLE version IS 'the version of the JDBC schema in use';
+    
+    ALTER TABLE polaris_schema.entities ADD COLUMN IF NOT EXISTS location_without_scheme TEXT;
     ```
+    - Please don't enable [OPTIMIZED_SIBLING_CHECK](https://github.com/apache/polaris/blob/740993963cb41c2c1b4638be5e04dd00f1263c98/polaris-core/src/main/java/org/apache/polaris/core/config/FeatureConfiguration.java#L346) feature configuration, once the above SQL statements are run. As it may lead to incorrect behavior, due to missing data for location_without_scheme column.
 - **Deprecations**
   - The property `polaris.active-roles-provider.type` is deprecated for removal.
   - The `ActiveRolesProvider` interface is deprecated for removal.

Makefile

@@ -144,7 +144,7 @@ client-unit-test: client-setup-env ## Run client unit tests
 	@echo "--- Client unit tests complete ---"
 
 .PHONY: client-integration-test
-client-integration-test: client-setup-env ## Run client integration tests
+client-integration-test: build-server client-setup-env ## Run client integration tests
 	@echo "--- Starting client integration tests ---"
 	@echo "Ensuring Docker Compose services are stopped and removed..."
 	@$(DOCKER) compose -f $(PYTHON_CLIENT_DIR)/docker-compose.yml kill || true # `|| true` prevents make from failing if containers don't exist

README.md

@@ -175,16 +175,33 @@ Default configuration values can be found in `runtime/defaults/src/main/resource
 
 #### Publishing Build Scans to develocity.apache.org
 
-All authenticated builds of Apache Polaris will automatically publish build scans to the ASF Develocity instance at 
-[develocity.apache.org](https://develocity.apache.org/scans?search.rootProjectNames=polaris). 
-
-CI builds originating from the `apache/polaris` repository will have access to the Apache organization-level secret 
-`DEVELOCITY_ACCESS_KEY` and publish build scans using the secret. CI builds originating from pull requests from forks 
-will not have access to the secret and will silently skip build scan publication.
-
-Apache committers can publish build scans from their local machine by 
-[provisioning an access key](https://docs.gradle.com/develocity/gradle-plugin/current/#automated_access_key_provisioning) 
-using ASF LDAP credentials. Builds by anonymous, unauthenticated contributors will silently skip build scan publication.
+Build scans of CI builds from a branch or tag in the `apache/polaris` repository on GitHub publish build scans
+to the ASF Develocity instance at
+[develocity.apache.org](https://develocity.apache.org/scans?search.rootProjectNames=polaris), if the workflow runs have access to the Apache organization-level secret 
+`DEVELOCITY_ACCESS_KEY`.
+
+Build scans of local developer builds publish build scans only if the Gradle command line option `--scan` is used.
+Those build scans are published to Gradle's public Develocity instance (see advanced configuration options below).
+Note that build scans on Gradle's public Develocity instance are publicly accessible to anyone.
+You have to accept Gradle's terms of service to publish to the Gradle's public Develocity instance.
+
+CI builds originating from pull requests against the `apache/polaris` GitHub repository are published to Gradle's
+_public_ Develocity instance. 
+
+Other CI build scans do only publish build scans to the Gradle's _public_ Develocity instance, if the environment
+variable `GRADLE_TOS_ACCEPTED` is set to `true`.
+By setting this variable you agree to the [Gradle's terms of service](https://gradle.com/terms-of-service), because
+accepting these ToS is your personal decision. 
+You can configure this environment variable for your GitHub repository in the GitHub repository settings under
+`Secrets` > `Secrets and variables` > `Actions` > choose the `Variables` tab > `New repository variable`. 
+
+Advanced configuration options for publishing build scans (only local and non-`apache/polaris` repository CI):
+* The project ID published with the build scan can be specified using the environment variable `DEVELOCITY_PROJECT_ID`.
+  The project ID defaults to the GitHub repository owner/name, for example `octocat/polaris`.
+* The Develocity server can be specified using the environment variable `DEVELOCITY_SERVER` if build scans should be
+  published to another than Gradle's public Develocity instance.
+* If you have to publish build scans to your own Develocity instance, you can configure the access key using a
+  GitHub secret named `DEVELOCITY_ACCESS_KEY`.
 
 ## License