Support HMS Federation #2355
Merged
Support HMS Federation #2355
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
poojanilangekar
force-pushed
the
hive-federation
branch
from
efa4a18 to
bffb367
Compare
Contributor
Author
Contributor
eric-maynard
left a comment
eric-maynard
left a comment
There was a problem hiding this comment.
Can we get a regression test like the one in #2286?
Contributor
Author
|
Yes, I have a test that I used locally. Unfortunately, I just can't add it to the test until we get this PR baked into the Docker image. :( |
Contributor
Author
|
Here is a jupyter notebook I've used in the past to demo the feature. (in lieu of the regtests that I will add after this PR is merged). |
eric-maynard
previously approved these changes
Aug 15, 2025
github-project-automation
bot
moved this from PRs In Progress
to Ready to merge
in Basic Kanban Board
dennishuo
previously approved these changes
Aug 15, 2025
Contributor
dennishuo
left a comment
dennishuo
left a comment
There was a problem hiding this comment.
LGTM, though is this the case where we wanted compile-time flags to allow the Polaris user to choose to exclude the whole org.apache.hive:hive-metastore dependency, even if we chose not to use a compile-time flag for HadoopCatalog due to that being a breaking change?
| String warehouse = ((HiveConnectionConfigInfoDpo) connectionConfigInfoDpo).getWarehouse(); | ||
| // Unlike Hadoop, HiveCatalog does not require us to create a Configuration object, the iceberg | ||
| // rest library find the default configuration by reading hive-site.xml in the classpath | ||
| // (including HADOOP_CONF_DIR classpath). |
Contributor
There was a problem hiding this comment.
Could be worth adding a TODO here to better qualify the assertion about not requiring Configuration and requiring hive-site.xml (and so we don't forget), that if we do want Hadoop Configuration to come from runtime Polaris properties instead, that we need to call hiveCatalog.setConf before the call to hiveCatalog.initialize and our constructed Configuration should begin with new Configuration(false) to avoid loading the default hive-site.xml?
AFAICT doing those two actions should be sufficient in potential multi-catalog environments to at least prevent basic conf-leakage between HiveCatalog instances.
poojanilangekar
dismissed stale reviews from dennishuo and eric-maynard
via
99d0867
sfc-gh-dhuo
approved these changes
Aug 16, 2025
dennishuo
approved these changes
Aug 16, 2025
github-project-automation
bot
moved this from Ready to merge
to Done
in Basic Kanban Board
eric-maynard
added a commit
that referenced
this pull request
Aug 18, 2025
In #2369 Iceberg table federation was refactored around the new `IcebergRESTExternalCatalogFactory` type based on discussion in the community sync. This has unblocked the ability to federate to more non-Iceberg catalogs, such as in #2355. This PR refactors generic table federation to go through the same mechanism. After this, we can go through and implement generic table federation for the existing `IcebergRESTExternalCatalogFactory` implementations.
bmlyr
pushed a commit
to bmlyr/polaris
that referenced
this pull request
Aug 19, 2025
In apache#2369 Iceberg table federation was refactored around the new `IcebergRESTExternalCatalogFactory` type based on discussion in the community sync. This has unblocked the ability to federate to more non-Iceberg catalogs, such as in apache#2355. This PR refactors generic table federation to go through the same mechanism. After this, we can go through and implement generic table federation for the existing `IcebergRESTExternalCatalogFactory` implementations.
snazy
added a commit
to snazy/polaris
that referenced
this pull request
Nov 20, 2025
* Refactor Authenticator and PolarisPrincipal (apache#2307) The main goal of this change is to facilitate future integration of federated principals: - `AuthenticatedPolarisPrincipal` becomes an interface `PolarisPrincipal`, as the original class leaks implementation details (references to `PrincipalEntity` and thus to the storage layer). The new interface does not reference the storage layer. This is one step further towards easy pluggability of authentication in Polaris. - The `Authenticator.authenticate()` method does not return an `Optional` anymore, as this was ambiguous (returning `Optional.empty()` vs throwing `NotAuthorizedException`). - Also the `Authenticator` interface is not generic anymore. This was an artifact of times when there were two kinds of `Authenticators` in Polaris (one for internal auth, the other for external) and is not necessary anymore. * Add PolarisDiagnostics field to TransactionalMetaStoreManagerImpl (apache#2361) the ultimate goal is removing the PolarisCallContext parameter from every PolarisMetaStoreManager interface method, so we make steps towards reducing its usage first. * Support HMS Federation (apache#2355) Supports federating to HiveCatalog using the Iceberg REST library. All hive dependencies are added in an independent module, i.e., `polaris-extensions-federation-hive` and can be removed/converted to a compile time flag if necessary. Similar to HadoopCatalog, HMS federation support is currently restricted to `IMPLICIT` auth. The underlying authentication can be any form that Hive supports, however Polaris will not store and manage any of these credentials. Again, similar to HadoopCatalog, this version supports federating to a single Hive instance. This PR relies on Polaris discovering the `hive-site.xml` file to get the configuration options from the classpath (including `HADOOP_CONF_DIR`). The spec change has been discussed in the [dev mailing list](https://lists.apache.org/thread/5qktjv6rzd8pghcl6f4oohko798o2p2g), followed by a discussion in the Polaris community sync on Aug 7, 2025. Testing: Modified the regression test to locally test that Hive federation works as expected. The next step would be to add a regression test once the change is baked into the Polaris docker image (for CI builds). This PR primarily builds on apache#1305 and apache#1466. Thank you @dennishuo and @eric-maynard for helping out with this! * Add PolarisDiagnostics field to TransactionWorkspaceMetaStoreManager (apache#2359) the ultimate goal is removing the `PolarisCallContext` parameter from every `PolarisMetaStoreManager` interface method, so we make steps towards reducing its usage first. * Rat-ignore user-settings for hugo-run-in-docker (apache#2376) * Modularize generic table federation (apache#2379) In apache#2369 Iceberg table federation was refactored around the new `IcebergRESTExternalCatalogFactory` type based on discussion in the community sync. This has unblocked the ability to federate to more non-Iceberg catalogs, such as in apache#2355. This PR refactors generic table federation to go through the same mechanism. After this, we can go through and implement generic table federation for the existing `IcebergRESTExternalCatalogFactory` implementations. * Update community meeting dates (apache#2382) * Reduce getRealmConfig calls (apache#2337) Classes with a `CallContext` field should call `getRealmConfig` once and store it as a field as well. The idea is that long term we would want to stop relying on the `CallContext` itself but instead inject its individual items. Thus we also add `RealmConfig` to `TestServices`. * Python client: make S3 role-ARN optional and add missing endpoint-internal property (apache#2339) * fix(deps): update dependency io.prometheus:prometheus-metrics-exporter-servlet-jakarta to v1.4.1 (apache#2377) * chore(deps): bump s3mock from 3.11.0 to 4.7.0 (apache#2375) Updates S3Mock testcontainer dependency from 3.11.0 to 4.7.0 and refactors usage into a centralized wrapper class in runtime/test-common. Changes Upgraded S3Mock testcontainer to 4.7.0 Created S3Mock wrapper class for consistent configuration Consolidated S3 config properties generation Updated integration tests to use new wrapper No functional changes to test behavior. * Nit: extract getResolvedCatalogEntity method in IcebergCatalogHandler (apache#2387) * Nit: remove transitive dependencies from runtime/server/build.gradle.kts (apache#2385) * Nit: add methods isExternal and isStaticFacade to CatalogEntity (apache#2386) * Minor refactor of integration test classes (apache#2384) This change promotes `CatalogConfig` and `RestCatalogConfig` to top-level, public annotations and introduces a few "hooks" in `PolarisRestCatalogIntegrationBase` that can be overridden by subclasses. This change is a preparatory work for apache#2280 (S3 remote signing). * Remove BaseMetaStoreManager.serializeProperties (apache#2374) similar to 7af85be we should prefer the existing helper methods on the entity instead * fix: minor corrections of documentation (apache#2397) - fixed dead link to catalog definition in Iceberg docs on Entities page - removed single quotes from credential parameter in the cmdline example for connecting a local spark-sql: env variables need to be resolved in cmdline, they will not be resolved by spark-sql itself. * chore(deps): update azure/setup-helm action to v4.3.1 (apache#2402) * Add 1.0.1 release to the website (apache#2400) * Add PolarisDiagnostics field to AbstractTransactionalPersistence (apache#2372) The ultimate goal is removing the `PolarisCallContext` parameter from every `PolarisMetaStoreManager` interface method, so we make steps towards reducing its usage first. * NoSQL: javadoc nit * Last merged commit fcd4777 --------- Co-authored-by: Alexandre Dutra <adutra@apache.org> Co-authored-by: Christopher Lambert <xn137@gmx.de> Co-authored-by: Pooja Nilangekar <poojan@umd.edu> Co-authored-by: Eric Maynard <eric.maynard+oss@snowflake.com> Co-authored-by: JB Onofré <jbonofre@apache.org> Co-authored-by: Mend Renovate <bot@renovateapp.com> Co-authored-by: Artur Rakhmatulin <from_github@binaryc.at> Co-authored-by: olsoloviov <40199597+olsoloviov@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Supports federating to HiveCatalog using the Iceberg REST library.
All hive dependencies are added in an independent module, i.e.,
polaris-extensions-federation-hiveand can be removed/converted to a compile time flag if necessary.Similar to HadoopCatalog, HMS federation support is currently restricted to
IMPLICITauth. The underlying authentication can be any form that Hive supports, however Polaris will not store and manage any of these credentials. Again, similar to HadoopCatalog, this version supports federating to a single Hive instance.This PR relies on Polaris discovering the
hive-site.xmlfile to get the configuration options from the classpath (includingHADOOP_CONF_DIR).The spec change has been discussed in the dev mailing list, followed by a discussion in the Polaris community sync on Aug 7, 2025.
Testing:
Modified the regression test to locally test that Hive federation works as expected. The next step would be to add a regression test once the change is baked into the Polaris docker image (for CI builds).
This PR primarily builds on #1305 and #1466. Thank you @dennishuo and @eric-maynard for helping out with this!