Dremio merge 2025 09 22 17 05 (apache#123)

* Suppress deprecation warnings in `PolarisSparkCatalog.createTable()` (apache#2631)

Background: apache#2394

Since we have to override the deprecated `createTable` method, we
suppress deprecation warnings produced by `javac`.

Suppressing `RedundantSuppression` is needed for IntelliJ, which
appears to consider this a normal situation and does not issue a
deprecation warning.

* Service: Add Events for PolarisServiceImpl APIs (apache#2482)

* CHANGELOG: Freeze change log for 1.1 and clear out unreleased version (apache#2635)

* Re-add CHANGELOG.md entry for apache#2197 (apache#2638)

Using `git log -p apache-polaris-1.1.0-incubating..553cb06 -- CHANGELOG.md` to find changes missed in the previous CHANGELOG update (apache#2635)

* Azure: Fix azure expires at prefix for the credentials refresh (apache#2633)

* Remove unused LOG in SparkCatalog (apache#2639)

* fix(deps): update dependency com.google.errorprone:error_prone_core to v2.42.0 (apache#2636)

* fix(deps): update dependency io.smallrye.config:smallrye-config-core to v3.14.0 (apache#2637)

* Fix client license check (apache#2642)

* fix(deps): update dependency software.amazon.awssdk:bom to v2.34.0 (apache#2645)

* fix(deps): update mockito monorepo to v5.20.0 (apache#2641)

* chore(deps): update docker.io/prom/prometheus docker tag to v3.6.0 (apache#2644)

* chore(events): unify in-memory buffer listeners implementations (apache#2628)

* fix(deps): update quarkus platform and group (apache#2595)

* Update jandex dependency to 3.5.0 (apache#2649)

* Last merged commit e6796f7

---------

Co-authored-by: Dmitri Bourlatchkov <dmitri.bourlatchkov@gmail.com>
Co-authored-by: Adnan Hemani <adnan.h@berkeley.edu>
Co-authored-by: Prashant Singh <35593236+singhpk234@users.noreply.github.com>
Co-authored-by: Mend Renovate <bot@renovateapp.com>
Co-authored-by: Yong Zheng <yongzheng0809@gmail.com>
Co-authored-by: Alexandre Dutra <adutra@apache.org>

Author: 7 people

CHANGELOG.md

@@ -29,84 +29,74 @@ request adding CHANGELOG notes for breaking (!) changes and possibly other secti
 
 ### Highlights
 
-- **HMS Federation Support**: Added support for Hive Metastore (HMS) federation, enabling integration with existing Hive metastores.
-
-- **Modularized Federation**: Introduced modularized federation architecture to support multiple catalog types and improve extensibility.
-
-- **External Authentication**: Added comprehensive support for external identity providers including Keycloak integration and Helm chart configuration options.
-
-- **Python Client Distribution**: The Python client is now packaged and distributed as a proper Python package for easier installation and usage.
-
-- **Catalog Federation CLI**: Extended the CLI with support for managing federated catalogs, making it easier to configure and operate catalog federation.
-
-- **MinIO**: Added MinIO integration support with comprehensive getting started documentation.
-
 ### Upgrade Notes
 
 - The EclipseLink Persistence implementation has been deprecated since 1.0.0 and will be completely removed
   in 1.3.0 or in 2.0.0 (whichever happens earlier).
 
 ### Breaking Changes
 
-- Helm chart: the default value of the `authentication.tokenBroker.secret.symmetricKey.secretKey` property has changed
-  from `symmetric.pem` to `symmetric.key`.
-
 ### New Features
 
-- Added Catalog configuration for S3 and STS endpoints. This also allows using non-AWS S3 implementations.
-  The realm-level feature flag `ALLOW_SETTING_S3_ENDPOINTS` (default: true) may be used to disable this
-  functionality.
-
-- The `IMPLICIT` authentication type enables users to create federated catalogs without explicitly
-providing authentication parameters to Polaris. When the authentication type is set to `IMPLICIT`,
-the authentication parameters are picked from the environment or configuration files.
-
-- The `DEFAULT_LOCATION_OBJECT_STORAGE_PREFIX_ENABLED` feature was added to support placing tables
-at locations that better optimize for object storage.
-
-- The `LIST_PAGINATION_ENABLED` (default: false) feature flag can be used to enable pagination
-  in the Iceberg REST Catalog API.
-
-- The Helm chart now supports Pod Disruption Budgets (PDBs) for Polaris components. This allows users to define
-  the minimum number of pods that must be available during voluntary disruptions, such as node maintenance.
-
-- Feature configuration `PURGE_VIEW_METADATA_ON_DROP` was added to allow dropping views without purging their metadata files.
-
-- Introduced S3 path-style access support for improved compatibility with S3-compatible storage systems.
-
-- Enhanced Python client with integration tests and improved error handling.
-
-- Introduced extensible pagination token implementation for better API performance.
-
-- Added support for `s3a` scheme in addition to existing S3 schemes.
-
-- Enhanced Helm chart with support for external authentication configuration and relational JDBC backend options.
-
-- Added comprehensive diagnostics and monitoring capabilities throughout the system.
-
-- Introduced bootstrap command options to specify custom schema files for database initialization.
-
-- Added refresh credentials endpoint configuration to LoadTableResponse for AWS, Azure, and GCP. Enabling
-automatic storage credential refresh per table on the client side. Java client version >= 1.8.0 is required.
-The endpoint path is always returned when using vended credentials, but clients must enable the 
-refresh-credentials flag for the desired storage provider.
-
 - Added a Management API endpoint to reset principal credentials, controlled by the `ENABLE_CREDENTIAL_RESET` (default: true) feature flag.
 
 ### Changes
 
-- Polaris Management API clients must be prepared to deal with new attributes in `AwsStorageConfigInfo` objects.
-
-- S3 configuration property role-ARN is no longer mandatory.
-
 ### Deprecations
 
 * The property `polaris.active-roles-provider.type` is deprecated and has no effect anymore.
 
 ### Fixes
 
+* Fixed incorrect Azure expires at field for the credentials refresh response, leading to client failure via #2633
+
 ### Commits
 
+## [1.1.0-incubating]
+Apache Polaris 1.1.0-incubating was released on September 19th, 2025.
+- **Highlights**
+  - **HMS Federation Support**: Added support for Hive Metastore (HMS) federation, enabling integration with existing Hive metastores.
+  - **Modularized Federation**: Introduced modularized federation architecture to support multiple catalog types and improve extensibility.
+  - **External Authentication**: Added comprehensive support for external identity providers including Keycloak integration and Helm chart configuration options.
+  - **Python Client Distribution**: The Python client is now packaged and distributed as a proper Python package for easier installation and usage.
+  - **Catalog Federation CLI**: Extended the CLI with support for managing federated catalogs, making it easier to configure and operate catalog federation.
+  - **MinIO**: Added MinIO integration support with comprehensive getting started documentation.
+- **New features**
+  - Added Catalog configuration for S3 and STS endpoints. This also allows using non-AWS S3 implementations.
+  - The realm-level feature flag `ALLOW_SETTING_S3_ENDPOINTS` (default: true) may be used to disable this
+    functionality.
+  - The `IMPLICIT` authentication type enables users to create federated catalogs without explicitly
+    providing authentication parameters to Polaris. When the authentication type is set to `IMPLICIT`,
+    the authentication parameters are picked from the environment or configuration files.
+  - The `DEFAULT_LOCATION_OBJECT_STORAGE_PREFIX_ENABLED` feature was added to support placing tables
+    at locations that better optimize for object storage.
+  - The `LIST_PAGINATION_ENABLED` (default: false) feature flag can be used to enable pagination
+    in the Iceberg REST Catalog API.
+  - The Helm chart now supports Pod Disruption Budgets (PDBs) for Polaris components. This allows users to define
+    the minimum number of pods that must be available during voluntary disruptions, such as node maintenance.
+  - Feature configuration `PURGE_VIEW_METADATA_ON_DROP` was added to allow dropping views without purging their metadata files.
+  - Introduced S3 path-style access support for improved compatibility with S3-compatible storage systems.
+  - Enhanced Python client with integration tests and improved error handling.
+  - Introduced extensible pagination token implementation for better API performance.
+  - Added support for `s3a` scheme in addition to existing S3 schemes.
+  - Enhanced Helm chart with support for external authentication configuration and relational JDBC backend options.
+  - Added comprehensive diagnostics and monitoring capabilities throughout the system.
+  - Introduced bootstrap command options to specify custom schema files for database initialization.
+  - Added refresh credentials endpoint configuration to LoadTableResponse for AWS, Azure, and GCP. Enabling
+    automatic storage credential refresh per table on the client side. Java client version >= 1.8.0 is required.
+    The endpoint path is always returned when using vended credentials, but clients must enable the
+    refresh-credentials flag for the desired storage provider.
+  - Added a Management API endpoint to reset principal credentials, controlled by the `ENABLE_CREDENTIAL_RESET` (default: true) feature flag.
+- **Changes**
+  - Polaris Management API clients must be prepared to deal with new attributes in `AwsStorageConfigInfo` objects.
+  - S3 configuration property role-ARN is no longer mandatory.
+- **Breaking changes**
+  - Helm chart: the default value of the `authentication.tokenBroker.secret.symmetricKey.secretKey` property has changed
+      from `symmetric.pem` to `symmetric.key`.
+- **Deprecations**
+  - The property `polaris.active-roles-provider.type` is deprecated for removal.
+  - The `ActiveRolesProvider` interface is deprecated for removal.
+
 ## [1.0.1-incubating]
 Apache Polaris 1.0.1-incubating was released on August 16th, 2025. It’s a maintenance release on the 1.0.0 release fixing a couple of issues on the Helm Chart:
 - remove db-kind in Helm Chart

client/python/pyproject.toml

@@ -62,40 +62,15 @@ mypy = ">=1.18, <=1.18.2"
 pyiceberg = "==0.10.0"
 pre-commit = "==4.3.0"
 openapi-generator-cli = "==7.11.0.post0"
-pip-licenses = "==5.0.0"
+pip-licenses-cli = "==v2.0.0"
 # pin virtualenv version to prevent poetry from upgrading to an incompatible version
 # see https://github.com/python-poetry/poetry/issues/10504#issuecomment-3176923981
 # 20.33.0 is the oldest version supported by poetry 2.2.0
 virtualenv = ">=20.33.0,<20.35.0"
 
 [tool.pip-licenses]
-from-classifier = true
-# Packages with "UNKNOWN" licenses in pip-licenses metadata.
-# These have been manually verified and are known to be compatible with ASF.
-ignore-packages = [
-    "anyio",                 # MIT License (MIT)
-    "build",                 # MIT License (MIT)
-    "CacheControl",          # Apache-2.0
-    "cffi",                  # MIT License (MIT)
-    "click",                 # BSD-3-Clause
-    "cryptography",          # Apache-2.0 or BSD-3-Clause
-    "fsspec",                # BSD-3-Clause
-    "jaraco.functools",      # MIT License (MIT)
-    "jeepney",               # MIT License (MIT)
-    "more-itertools",        # MIT License (MIT)
-    "mypy_extensions",       # MIT License (MIT)
-    "pyparsing",             # MIT License (MIT)
-    "RapidFuzz",             # MIT License (MIT)
-    "SecretStorage",         # BSD-3-Clause
-    "types-python-dateutil", # Apache-2.0
-    "typing-inspection",     # MIT License (MIT)
-    "typing_extensions",     # PSF-2.0
-    "urllib3",               # MIT License (MIT)
-    "zipp",                  # MIT License (MIT)
-    "zstandard",             # BSD-3-Clause
-]
 partial-match = true
-allow-only = "MIT;Apache;BSD License;PSF-2.0;ISC;The Unlicense;Python Software Foundation License;Mozilla Public License"
+allow-only = "Apache;BSD License;BSD-3-Clause;ISC;MIT;Mozilla Public License;PSF-2.0;Python Software Foundation License;The Unlicense"
 
 [build-system]
 requires = ["poetry-core>=2.0.0,<3.0.0", "openapi-generator-cli==7.11.0.post0"]

getting-started/telemetry/docker-compose.yml

@@ -67,7 +67,7 @@ services:
     entrypoint: '/bin/sh -c "chmod +x /polaris/create-catalog.sh && /polaris/create-catalog.sh"'
 
   prometheus:
-    image: docker.io/prom/prometheus:v3.5.0
+    image: docker.io/prom/prometheus:v3.6.0
     ports:
       - "9093:9090"
     depends_on:

gradle/libs.versions.toml

@@ -22,7 +22,7 @@ checkstyle = "10.25.0"
 hadoop = "3.4.2"
 hive = "3.1.3"
 iceberg = "1.10.0" # Ensure to update the iceberg version in regtests to keep regtests up-to-date
-quarkus = "3.26.3"
+quarkus = "3.26.4"
 immutables = "2.11.3"
 jmh = "1.37"
 picocli = "4.7.7"
@@ -43,7 +43,7 @@ agrona = { module = "org.agrona:agrona", version = "2.2.4" }
 antlr4-runtime = { module = "org.antlr:antlr4-runtime", version.strictly = "4.9.3" } # spark integration tests
 assertj-core = { module = "org.assertj:assertj-core", version = "3.27.5" }
 auth0-jwt = { module = "com.auth0:java-jwt", version = "4.5.0" }
-awssdk-bom = { module = "software.amazon.awssdk:bom", version = "2.33.9" }
+awssdk-bom = { module = "software.amazon.awssdk:bom", version = "2.34.0" }
 awaitility = { module = "org.awaitility:awaitility", version = "4.3.0" }
 azuresdk-bom = { module = "com.azure:azure-sdk-bom", version = "1.2.38" }
 caffeine = { module = "com.github.ben-manes.caffeine:caffeine", version = "3.2.2" }
@@ -52,7 +52,7 @@ commons-lang3 = { module = "org.apache.commons:commons-lang3", version = "3.18.0
 commons-text = { module = "org.apache.commons:commons-text", version = "1.14.0" }
 docker-java-api = { module = "com.github.docker-java:docker-java-api", version = "3.5.3" }
 eclipselink = { module = "org.eclipse.persistence:eclipselink", version = "4.0.7" }
-errorprone = { module = "com.google.errorprone:error_prone_core", version = "2.41.0" }
+errorprone = { module = "com.google.errorprone:error_prone_core", version = "2.42.0" }
 google-cloud-storage-bom = { module = "com.google.cloud:google-cloud-storage-bom", version = "2.57.0" }
 guava = { module = "com.google.guava:guava", version = "33.5.0-jre" }
 h2 = { module = "com.h2database:h2", version = "2.3.232" }
@@ -75,7 +75,7 @@ jakarta-persistence-api = { module = "jakarta.persistence:jakarta.persistence-ap
 jakarta-servlet-api = { module = "jakarta.servlet:jakarta.servlet-api", version = "6.1.0" }
 jakarta-validation-api = { module = "jakarta.validation:jakarta.validation-api", version = "3.1.1" }
 jakarta-ws-rs-api = { module = "jakarta.ws.rs:jakarta.ws.rs-api", version = "4.0.0" }
-jandex = { module = "io.smallrye.jandex:jandex", version ="3.4.0" }
+jandex = { module = "io.smallrye.jandex:jandex", version ="3.5.0" }
 javax-servlet-api = { module = "javax.servlet:javax.servlet-api", version = "4.0.1" }
 jcstress-core = { module = "org.openjdk.jcstress:jcstress-core", version = "0.16" }
 jmh-core = { module = "org.openjdk.jmh:jmh-core", version.ref = "jmh" }
@@ -86,8 +86,8 @@ localstack = { module = "org.testcontainers:localstack", version = "1.21.3" }
 logback-classic = { module = "ch.qos.logback:logback-classic", version = "1.5.18" }
 micrometer-bom = { module = "io.micrometer:micrometer-bom", version = "1.15.4" }
 microprofile-fault-tolerance-api = { module = "org.eclipse.microprofile.fault-tolerance:microprofile-fault-tolerance-api", version = "4.1.2" }
-mockito-core = { module = "org.mockito:mockito-core", version = "5.19.0" }
-mockito-junit-jupiter = { module = "org.mockito:mockito-junit-jupiter", version = "5.19.0" }
+mockito-core = { module = "org.mockito:mockito-core", version = "5.20.0" }
+mockito-junit-jupiter = { module = "org.mockito:mockito-junit-jupiter", version = "5.20.0" }
 mongodb-driver-sync = { module = "org.mongodb:mongodb-driver-sync", version = "5.6.0" }
 opentelemetry-bom = { module = "io.opentelemetry:opentelemetry-bom", version = "1.54.1" }
 opentelemetry-instrumentation-bom-alpha = { module = "io.opentelemetry.instrumentation:opentelemetry-instrumentation-bom-alpha", version= "2.19.0-alpha" }
@@ -102,7 +102,7 @@ scala212-lang-reflect = { module = "org.scala-lang:scala-reflect", version.ref =
 s3mock-testcontainers = { module = "com.adobe.testing:s3mock-testcontainers", version = "4.9.1" }
 slf4j-api = { module = "org.slf4j:slf4j-api", version.ref = "slf4j" }
 smallrye-common-annotation = { module = "io.smallrye.common:smallrye-common-annotation", version = "2.13.9" }
-smallrye-config-core = { module = "io.smallrye.config:smallrye-config-core", version = "3.13.4" }
+smallrye-config-core = { module = "io.smallrye.config:smallrye-config-core", version = "3.14.0" }
 smallrye-jandex = { module = "io.smallrye:jandex", version = "3.4.0" }
 spark35-sql-scala212 = { module = "org.apache.spark:spark-sql_2.12", version.ref = "spark35" }
 swagger-annotations = { module = "io.swagger:swagger-annotations", version.ref = "swagger" }

plugins/spark/v3.5/spark/src/main/java/org/apache/polaris/spark/PolarisSparkCatalog.java

@@ -78,6 +78,7 @@ public Table loadTable(Identifier identifier) throws NoSuchTableException {
   }
 
   @Override
+  @SuppressWarnings({"deprecation", "RedundantSuppression"})
   public Table createTable(
       Identifier identifier,
       StructType schema,

plugins/spark/v3.5/spark/src/main/java/org/apache/polaris/spark/SparkCatalog.java

@@ -51,8 +51,6 @@
 import org.apache.spark.sql.connector.expressions.Transform;
 import org.apache.spark.sql.types.StructType;
 import org.apache.spark.sql.util.CaseInsensitiveStringMap;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
 
 /**
  * SparkCatalog Implementation that is able to interact with both Iceberg SparkCatalog and Polaris
@@ -66,7 +64,6 @@ public class SparkCatalog
         SupportsNamespaces,
         ViewCatalog,
         SupportsReplaceView {
-  private static final Logger LOG = LoggerFactory.getLogger(SparkCatalog.class);
 
   @VisibleForTesting protected String catalogName = null;
   @VisibleForTesting protected org.apache.iceberg.spark.SparkCatalog icebergsSparkCatalog = null;

polaris-core/src/main/java/org/apache/polaris/core/storage/StorageAccessProperty.java

@@ -19,6 +19,7 @@
 package org.apache.polaris.core.storage;
 
 import org.apache.iceberg.aws.AwsClientProperties;
+import org.apache.iceberg.azure.AzureProperties;
 import org.apache.iceberg.gcp.GCPProperties;
 
 /**
@@ -69,7 +70,7 @@ public enum StorageAccessProperty {
   AZURE_SAS_TOKEN(String.class, "adls.sas-token.", "an azure shared access signature token"),
   AZURE_REFRESH_CREDENTIALS_ENDPOINT(
       String.class,
-      "adls.refresh-credentials-endpoint",
+      AzureProperties.ADLS_REFRESH_CREDENTIALS_ENDPOINT,
       "the endpoint to load vended credentials for a table from the catalog",
       false,
       false),
@@ -78,6 +79,12 @@ public enum StorageAccessProperty {
       "expiration-time",
       "the expiration time for the access token, in milliseconds",
       true,
+      true),
+  AZURE_SAS_TOKEN_EXPIRES_AT_MS_PREFIX(
+      Long.class,
+      AzureProperties.ADLS_SAS_TOKEN_EXPIRES_AT_MS_PREFIX,
+      "The expiration time for the access token, in milliseconds",
+      true,
       true);
 
   private final Class valueType;

polaris-core/src/main/java/org/apache/polaris/core/storage/azure/AzureCredentialsStorageIntegration.java

@@ -182,7 +182,7 @@ static AccessConfig toAccessConfig(
       Instant expiresAt,
       Optional<String> refreshCredentialsEndpoint) {
     AccessConfig.Builder accessConfig = AccessConfig.builder();
-    handleAzureCredential(accessConfig, sasToken, storageDnsName);
+    handleAzureCredential(accessConfig, sasToken, storageDnsName, expiresAt);
     accessConfig.put(
         StorageAccessProperty.EXPIRATION_TIME, String.valueOf(expiresAt.toEpochMilli()));
     refreshCredentialsEndpoint.ifPresent(
@@ -193,8 +193,11 @@ static AccessConfig toAccessConfig(
   }
 
   private static void handleAzureCredential(
-      AccessConfig.Builder config, String sasToken, String host) {
+      AccessConfig.Builder config, String sasToken, String host, Instant expiresAt) {
     config.putCredential(StorageAccessProperty.AZURE_SAS_TOKEN.getPropertyName() + host, sasToken);
+    config.putCredential(
+        StorageAccessProperty.AZURE_SAS_TOKEN_EXPIRES_AT_MS_PREFIX.getPropertyName() + host,
+        String.valueOf(expiresAt.toEpochMilli()));
 
     // Iceberg 1.7.x may expect the credential key to _not_ be suffixed with endpoint
     if (host.endsWith(AzureLocation.ADLS_ENDPOINT)) {