Dremio merge 2025 08 20 13 44 (apache#105)

* Refactor Authenticator and PolarisPrincipal (apache#2307)

The main goal of this change is to facilitate future integration of federated principals:

- `AuthenticatedPolarisPrincipal` becomes an interface `PolarisPrincipal`, as the original class leaks implementation details (references to `PrincipalEntity` and thus to the storage layer). The new interface does not reference the storage layer. This is one step further towards easy pluggability of authentication in Polaris.

- The `Authenticator.authenticate()` method does not return an `Optional` anymore, as this was ambiguous (returning `Optional.empty()` vs throwing `NotAuthorizedException`).

- Also the `Authenticator` interface is not generic anymore. This was an artifact of times when there were two kinds of `Authenticators` in Polaris (one for internal auth, the other for external) and is not necessary anymore.

* Add PolarisDiagnostics field to TransactionalMetaStoreManagerImpl (apache#2361)

the ultimate goal is removing the PolarisCallContext parameter from every
PolarisMetaStoreManager interface method, so we make steps towards reducing
its usage first.

* Support HMS Federation (apache#2355)

Supports federating to HiveCatalog using the Iceberg REST library. 
All hive dependencies are added in an independent module, i.e., `polaris-extensions-federation-hive` and can be removed/converted to a compile time flag if necessary. 
Similar to HadoopCatalog, HMS federation support is currently restricted to `IMPLICIT` auth. The underlying authentication can be any form that Hive supports, however Polaris will not store and manage any of these credentials. Again, similar to HadoopCatalog, this version supports federating to a single Hive instance. 

This PR relies on Polaris discovering the `hive-site.xml` file to get the configuration options from the classpath (including `HADOOP_CONF_DIR`).

The spec change has been discussed in the [dev mailing list](https://lists.apache.org/thread/5qktjv6rzd8pghcl6f4oohko798o2p2g), followed by a discussion in the Polaris community sync on Aug 7, 2025. 

Testing: 
Modified the regression test to locally test that Hive federation works as expected. The next step would be to add a regression test once the change is baked into the Polaris docker image (for CI builds).


This PR primarily builds on apache#1305 and apache#1466. Thank you @dennishuo and @eric-maynard for helping out with this!

* Add PolarisDiagnostics field to TransactionWorkspaceMetaStoreManager (apache#2359)

the ultimate goal is removing the `PolarisCallContext` parameter from every
`PolarisMetaStoreManager` interface method, so we make steps towards
reducing its usage first.

* Rat-ignore user-settings for hugo-run-in-docker (apache#2376)

* Modularize generic table federation (apache#2379)

In apache#2369 Iceberg table federation was refactored around the new `IcebergRESTExternalCatalogFactory` type based on discussion in the community sync. This has unblocked the ability to federate to more non-Iceberg catalogs, such as in apache#2355. This PR refactors generic table federation to go through the same mechanism. After this, we can go through and implement generic table federation for the existing `IcebergRESTExternalCatalogFactory` implementations.

* Update community meeting dates (apache#2382)

* Reduce getRealmConfig calls (apache#2337)

Classes with a `CallContext` field should call `getRealmConfig` once and store it as a field as well.
The idea is that long term we would want to stop relying on the `CallContext` itself but instead inject its individual items. Thus we also add `RealmConfig` to `TestServices`.

* Python client: make S3 role-ARN optional and add missing endpoint-internal property (apache#2339)

* fix(deps): update dependency io.prometheus:prometheus-metrics-exporter-servlet-jakarta to v1.4.1 (apache#2377)

* chore(deps): bump s3mock from 3.11.0 to 4.7.0 (apache#2375)

Updates S3Mock testcontainer dependency from 3.11.0 to 4.7.0 and refactors usage into a centralized wrapper class in runtime/test-common.

Changes

    Upgraded S3Mock testcontainer to 4.7.0
    Created S3Mock wrapper class for consistent configuration
    Consolidated S3 config properties generation
    Updated integration tests to use new wrapper

No functional changes to test behavior.

* Nit: extract getResolvedCatalogEntity method in IcebergCatalogHandler (apache#2387)

* Nit: remove transitive dependencies from runtime/server/build.gradle.kts (apache#2385)

* Nit: add methods isExternal and isStaticFacade to CatalogEntity (apache#2386)

* Minor refactor of integration test classes (apache#2384)

This change promotes `CatalogConfig` and `RestCatalogConfig` to top-level, public annotations and introduces a few "hooks" in `PolarisRestCatalogIntegrationBase` that can be overridden by subclasses.

This change is a preparatory work for apache#2280 (S3 remote signing).

* Remove BaseMetaStoreManager.serializeProperties (apache#2374)

similar to 7af85be we should prefer
the existing helper methods on the entity instead

* fix: minor corrections of documentation (apache#2397)

- fixed dead link to catalog definition in Iceberg docs on Entities page
- removed single quotes from credential parameter in the cmdline example for connecting a local spark-sql: env variables need to be resolved in cmdline, they will not be resolved by spark-sql itself.

* chore(deps): update azure/setup-helm action to v4.3.1 (apache#2402)

* Add 1.0.1 release to the website (apache#2400)

* Add PolarisDiagnostics field to AbstractTransactionalPersistence (apache#2372)

The ultimate goal is removing the `PolarisCallContext` parameter from every `PolarisMetaStoreManager` interface method, so we make steps towards reducing its usage first.

* NoSQL: javadoc nit

* Last merged commit fcd4777

---------

Co-authored-by: Alexandre Dutra <adutra@apache.org>
Co-authored-by: Christopher Lambert <xn137@gmx.de>
Co-authored-by: Pooja Nilangekar <poojan@umd.edu>
Co-authored-by: Eric Maynard <eric.maynard+oss@snowflake.com>
Co-authored-by: JB Onofré <jbonofre@apache.org>
Co-authored-by: Mend Renovate <bot@renovateapp.com>
Co-authored-by: Artur Rakhmatulin <from_github@binaryc.at>
Co-authored-by: olsoloviov <40199597+olsoloviov@users.noreply.github.com>

Author: 9 people

.github/workflows/helm.yml

@@ -50,7 +50,7 @@ jobs:
           distribution: 'temurin'
 
       - name: Set up Helm
-        uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
+        uses: azure/setup-helm@1a275c3b69536ee54be43f2070a358922e12c8d4 # v4.3.1
         with:
           version: 'v3.16.0'
 

build.gradle.kts

@@ -109,6 +109,7 @@ tasks.named<RatTask>("rat").configure {
 
   // Web site
   excludes.add("**/go.sum")
+  excludes.add("site/.user-settings")
   excludes.add("site/node_modules/**")
   excludes.add("site/layouts/robots.txt")
   // Ignore generated stuff, when the Hugo is run w/o Docker

client/python/cli/command/__init__.py

@@ -66,6 +66,7 @@ def options_get(key, f=lambda x: x):
                 iceberg_remote_catalog_name=options_get(Arguments.ICEBERG_REMOTE_CATALOG_NAME),
                 remove_properties=[] if remove_properties is None else remove_properties,
                 endpoint=options_get(Arguments.ENDPOINT),
+                endpoint_internal=options_get(Arguments.ENDPOINT_INTERNAL),
                 sts_endpoint=options_get(Arguments.STS_ENDPOINT),
                 path_style_access=options_get(Arguments.PATH_STYLE_ACCESS),
                 catalog_connection_type=options_get(Arguments.CATALOG_CONNECTION_TYPE),

client/python/cli/command/catalogs.py

@@ -65,6 +65,7 @@ class CatalogsCommand(Command):
     hadoop_warehouse: str
     iceberg_remote_catalog_name: str
     endpoint: str
+    endpoint_internal: str
     sts_endpoint: str
     path_style_access: bool
     catalog_connection_type: str
@@ -121,18 +122,17 @@ def validate(self):
                                 f" {Argument.to_flag_name(Arguments.CATALOG_SERVICE_IDENTITY_IAM_ARN)}")
 
         if self.storage_type == StorageType.S3.value:
-            if not self.role_arn:
-                raise Exception(
-                    f"Missing required argument for storage type 's3':"
-                    f" {Argument.to_flag_name(Arguments.ROLE_ARN)}"
-                )
             if self._has_azure_storage_info() or self._has_gcs_storage_info():
                 raise Exception(
-                    f"Storage type 's3' supports the storage credentials"
+                    f"Storage type 's3' supports the options"
                     f" {Argument.to_flag_name(Arguments.ROLE_ARN)},"
                     f" {Argument.to_flag_name(Arguments.REGION)},"
-                    f" {Argument.to_flag_name(Arguments.EXTERNAL_ID)}, and"
-                    f" {Argument.to_flag_name(Arguments.USER_ARN)}"
+                    f" {Argument.to_flag_name(Arguments.EXTERNAL_ID)},"
+                    f" {Argument.to_flag_name(Arguments.USER_ARN)},"
+                    f" {Argument.to_flag_name(Arguments.ENDPOINT)},"
+                    f" {Argument.to_flag_name(Arguments.ENDPOINT_INTERNAL)},"
+                    f" {Argument.to_flag_name(Arguments.STS_ENDPOINT)}, and"
+                    f" {Argument.to_flag_name(Arguments.PATH_STYLE_ACCESS)}"
                 )
         elif self.storage_type == StorageType.AZURE.value:
             if not self.tenant_id:
@@ -142,7 +142,7 @@ def validate(self):
                 )
             if self._has_aws_storage_info() or self._has_gcs_storage_info():
                 raise Exception(
-                    "Storage type 'azure' supports the storage credentials"
+                    "Storage type 'azure' supports the options"
                     f" {Argument.to_flag_name(Arguments.TENANT_ID)},"
                     f" {Argument.to_flag_name(Arguments.MULTI_TENANT_APP_NAME)}, and"
                     f" {Argument.to_flag_name(Arguments.CONSENT_URL)}"
@@ -160,11 +160,11 @@ def validate(self):
                 or self._has_gcs_storage_info()
             ):
                 raise Exception(
-                    "Storage type 'file' does not support any storage credentials"
+                    "Storage type 'file' does not support any additional options"
                 )
 
     def _has_aws_storage_info(self):
-        return self.role_arn or self.external_id or self.user_arn or self.region or self.endpoint or self.sts_endpoint or self.path_style_access
+        return self.role_arn or self.external_id or self.user_arn or self.region or self.endpoint or self.endpoint_internal or self.sts_endpoint or self.path_style_access
 
     def _has_azure_storage_info(self):
         return self.tenant_id or self.multi_tenant_app_name or self.consent_url
@@ -183,6 +183,7 @@ def _build_storage_config_info(self):
                 user_arn=self.user_arn,
                 region=self.region,
                 endpoint=self.endpoint,
+                endpoint_internal=self.endpoint_internal,
                 sts_endpoint=self.sts_endpoint,
                 path_style_access=self.path_style_access,
             )

client/python/cli/constants.py

@@ -168,6 +168,7 @@ class Arguments:
     HADOOP_WAREHOUSE = "hadoop_warehouse"
     ICEBERG_REMOTE_CATALOG_NAME = "iceberg_remote_catalog_name"
     ENDPOINT = "endpoint"
+    ENDPOINT_INTERNAL = "endpoint_internal"
     STS_ENDPOINT = "sts_endpoint"
     PATH_STYLE_ACCESS = "path_style_access"
     CATALOG_CONNECTION_TYPE = "catalog_connection_type"
@@ -223,11 +224,12 @@ class Create:
                 "Multiple locations can be provided by specifying this option more than once."
             )
 
-            ROLE_ARN = "(Required for S3) A role ARN to use when connecting to S3"
+            ROLE_ARN = "(Only for S3) A role ARN to use when connecting to S3"
             EXTERNAL_ID = "(Only for S3) The external ID to use when connecting to S3"
             REGION = "(Only for S3) The region to use when connecting to S3"
             USER_ARN = "(Only for S3) A user ARN to use when connecting to S3"
             ENDPOINT = "(Only for S3) The S3 endpoint to use when connecting to S3"
+            ENDPOINT_INTERNAL = "(Only for S3) The S3 endpoint used by Polaris to use when connecting to S3, if different from the one that clients use"
             STS_ENDPOINT = (
                 "(Only for S3) The STS endpoint to use when connecting to STS"
             )

client/python/cli/options/option_tree.py

@@ -117,6 +117,7 @@ def get_tree() -> List[Option]:
                              choices=[st.value for st in StorageType]),
                     Argument(Arguments.DEFAULT_BASE_LOCATION, str, Hints.Catalogs.Create.DEFAULT_BASE_LOCATION),
                     Argument(Arguments.ENDPOINT, str, Hints.Catalogs.Create.ENDPOINT),
+                    Argument(Arguments.ENDPOINT_INTERNAL, str, Hints.Catalogs.Create.ENDPOINT_INTERNAL),
                     Argument(Arguments.STS_ENDPOINT, str, Hints.Catalogs.Create.STS_ENDPOINT),
                     Argument(Arguments.PATH_STYLE_ACCESS, bool, Hints.Catalogs.Create.PATH_STYLE_ACCESS),
                     Argument(Arguments.ALLOWED_LOCATION, str, Hints.Catalogs.Create.ALLOWED_LOCATION,

extensions/federation/hadoop/src/main/java/org/apache/polaris/extensions/federation/hadoop/HadoopFederatedCatalogFactory.java

@@ -24,6 +24,7 @@
 import org.apache.iceberg.catalog.Catalog;
 import org.apache.iceberg.hadoop.HadoopCatalog;
 import org.apache.polaris.core.catalog.ExternalCatalogFactory;
+import org.apache.polaris.core.catalog.GenericTableCatalog;
 import org.apache.polaris.core.connection.AuthenticationParametersDpo;
 import org.apache.polaris.core.connection.AuthenticationType;
 import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
@@ -58,4 +59,12 @@ public Catalog createCatalog(
         warehouse, connectionConfigInfoDpo.asIcebergCatalogProperties(userSecretsManager));
     return hadoopCatalog;
   }
+
+  @Override
+  public GenericTableCatalog createGenericCatalog(
+      ConnectionConfigInfoDpo connectionConfig, UserSecretsManager userSecretsManager) {
+    // TODO implement
+    throw new UnsupportedOperationException(
+        "Generic table federation to this catalog is not supported.");
+  }
 }

extensions/federation/hive/README.md

@@ -0,0 +1,31 @@
+<!--
+  Licensed to the Apache Software Foundation (ASF) under one
+  or more contributor license agreements.  See the NOTICE file
+  distributed with this work for additional information
+  regarding copyright ownership.  The ASF licenses this file
+  to you under the Apache License, Version 2.0 (the
+  "License"); you may not use this file except in compliance
+  with the License.  You may obtain a copy of the License at
+ 
+   http://www.apache.org/licenses/LICENSE-2.0
+ 
+  Unless required by applicable law or agreed to in writing,
+  software distributed under the License is distributed on an
+  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  KIND, either express or implied.  See the License for the
+  specific language governing permissions and limitations
+  under the License.
+-->
+### Using the `HiveFederatedCatalogFactory`
+
+This `HiveFederatedCatalogFactory` module is an independent compilation unit and will be built into the Polaris binary only when the following flag is set in the gradle.properties file:
+```
+NonRESTCatalogs=HIVE,<alternates>
+```
+
+The other option is to pass it as an argument to the gradle JVM as follows: 
+```
+./gradlew build -DNonRESTCatalogs=HIVE
+```
+
+Without this flag, the Hive factory won't be compiled into Polaris and therefore Polaris will not load the class at runtime, throwing an unsupported exception for federated catalog calls.

extensions/federation/hive/build.gradle.kts

@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+plugins {
+  id("polaris-client")
+  alias(libs.plugins.jandex)
+}
+
+dependencies {
+  // Polaris dependencies
+  implementation(project(":polaris-core"))
+
+  implementation(platform(libs.iceberg.bom))
+  implementation("org.apache.iceberg:iceberg-api")
+  implementation("org.apache.iceberg:iceberg-core")
+  implementation("org.apache.iceberg:iceberg-common")
+  // Use iceberg-hive-metastore but exclude conflicting hive dependencies
+  implementation("org.apache.iceberg:iceberg-hive-metastore") { exclude(group = "org.apache.hive") }
+  // Add our own Hive 4.1.0 dependencies
+  implementation(libs.hive.metastore) {
+    exclude("org.slf4j", "slf4j-reload4j")
+    exclude("org.slf4j", "slf4j-log4j12")
+    exclude("ch.qos.reload4j", "reload4j")
+    exclude("log4j", "log4j")
+    exclude("org.apache.zookeeper", "zookeeper")
+  }
+
+  // Hadoop dependencies
+  implementation(libs.hadoop.common) {
+    exclude("org.slf4j", "slf4j-reload4j")
+    exclude("org.slf4j", "slf4j-log4j12")
+    exclude("ch.qos.reload4j", "reload4j")
+    exclude("log4j", "log4j")
+    exclude("org.apache.zookeeper", "zookeeper")
+    exclude("org.apache.hadoop.thirdparty", "hadoop-shaded-protobuf_3_25")
+    exclude("com.github.pjfanning", "jersey-json")
+    exclude("com.sun.jersey", "jersey-core")
+    exclude("com.sun.jersey", "jersey-server")
+    exclude("com.sun.jersey", "jersey-servlet")
+    exclude("io.dropwizard.metrics", "metrics-core")
+  }
+
+  // CDI dependencies for runtime discovery
+  implementation(libs.jakarta.enterprise.cdi.api)
+  implementation(libs.smallrye.common.annotation)
+
+  // Logging
+  implementation(libs.slf4j.api)
+}

extensions/federation/hive/src/main/java/org/apache/polaris/extensions/federation/hive/HiveFederatedCatalogFactory.java

@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.extensions.federation.hive;
+
+import io.smallrye.common.annotation.Identifier;
+import jakarta.enterprise.context.ApplicationScoped;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.hive.HiveCatalog;
+import org.apache.polaris.core.catalog.ExternalCatalogFactory;
+import org.apache.polaris.core.catalog.GenericTableCatalog;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.AuthenticationType;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import org.apache.polaris.core.connection.hive.HiveConnectionConfigInfoDpo;
+import org.apache.polaris.core.secrets.UserSecretsManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** Factory class for creating a Hive catalog handle based on connection configuration. */
+@ApplicationScoped
+@Identifier(ConnectionType.HIVE_FACTORY_IDENTIFIER)
+public class HiveFederatedCatalogFactory implements ExternalCatalogFactory {
+  private static final Logger LOGGER = LoggerFactory.getLogger(HiveFederatedCatalogFactory.class);
+
+  @Override
+  public Catalog createCatalog(
+      ConnectionConfigInfoDpo connectionConfigInfoDpo, UserSecretsManager userSecretsManager) {
+    // Currently, Polaris supports Hive federation only via IMPLICIT authentication.
+    // Hence, prior to initializing the configuration, ensure that the catalog uses
+    // IMPLICIT authentication.
+    AuthenticationParametersDpo authenticationParametersDpo =
+        connectionConfigInfoDpo.getAuthenticationParameters();
+    if (authenticationParametersDpo.getAuthenticationTypeCode()
+        != AuthenticationType.IMPLICIT.getCode()) {
+      throw new IllegalStateException("Hive federation only supports IMPLICIT authentication.");
+    }
+    String warehouse = ((HiveConnectionConfigInfoDpo) connectionConfigInfoDpo).getWarehouse();
+    // Unlike Hadoop, HiveCatalog does not require us to create a Configuration object, the iceberg
+    // rest library find the default configuration by reading hive-site.xml in the classpath
+    // (including HADOOP_CONF_DIR classpath).
+
+    // TODO: In the future, we could support multiple HiveCatalog instances based on polaris/catalog
+    // properties.
+    // A brief set of setps involved (and the options):
+    // 1. Create a configuration without default properties.
+    //  `Configuration conf = new Configuration(boolean loadDefaults=false);`
+    // 2a. Specify the hive-site.xml file path in the configuration.
+    //  `conf.addResource(new Path(hiveSiteXmlPath));`
+    // 2b. Specify individual properties in the configuration.
+    //  `conf.set(property, value);`
+    // Polaris could support federating to multiple LDAP based Hive metastores. Multiple
+    // Kerberos instances are not suitable because Kerberos ties a single identity to the server.
+    HiveCatalog hiveCatalog = new HiveCatalog();
+    hiveCatalog.initialize(
+        warehouse, connectionConfigInfoDpo.asIcebergCatalogProperties(userSecretsManager));
+    return hiveCatalog;
+  }
+
+  @Override
+  public GenericTableCatalog createGenericCatalog(
+      ConnectionConfigInfoDpo connectionConfig, UserSecretsManager userSecretsManager) {
+    // TODO implement
+    throw new UnsupportedOperationException(
+        "Generic table federation to this catalog is not supported.");
+  }
+}