Fix Hadoop federation to initialize the configuration #2282
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
Author
|
CC : @eric-maynard |
poojanilangekar
force-pushed
the
hadoop-integration
branch
from
fcf760d to
ebef8e2
Compare
poojanilangekar
force-pushed
the
hadoop-integration
branch
from
ebef8e2 to
26e479f
Compare
eric-maynard
approved these changes
Aug 6, 2025
github-project-automation
bot
moved this from PRs In Progress
to Ready to merge
in Basic Kanban Board
github-project-automation
bot
moved this from Ready to merge
to Done
in Basic Kanban Board
snazy
added a commit
to snazy/polaris
that referenced
this pull request
Nov 20, 2025
* Helm Chart: remove duplicate line in authenticationOptions template (apache#2297) * Move python client Makefile into the root level one (apache#2140) * Move python client Makefile into the root level one * Update workflow * add client-lint to pre-commit * Update README.md to include client * fix(deps): update dependency boto3 to v1.40.5 (apache#2300) * fix(deps): update dependency org.assertj:assertj-core to v3.27.4 (apache#2298) * Fix python CI client-integration-test (apache#2305) `CONTAINER_TOOL` got renamed to `DOCKER` in 6764a88 but then eda7074 added more `CONTAINER_TOOL` usage * fix(docs): update BasePolarisAuthenticator to DefaultAuthenticator (apache#2303) * Fix Hadoop federation to initialize the configuration prior to catalog initialization (apache#2282) This PR addresses the gap in Hadoop federation to ensure that we initialize the hadoop configuration prior to initializing the catalog object. The iceberg library expects the hadoop configuration to be initialized before creating a HadoopCatalog object. This change ensures that Polaris is compatible with the underlying iceberg library. Additionally, since the config initialization is based on the underlying (default) core-site.xml file, the change ensures that the federated catalog was created using `IMPLICIT` authentication mode. Testing: Due to current limitations in the current test setup, tested the change manually. [TODO] Add a regtest with hadoop federation once the change is baked into the apache/polaris docker image. * fix(deps): update dependency boto3 to v1.40.6 (apache#2308) * fix(deps): update dependency io.opentelemetry:opentelemetry-bom to v1.53.0 (apache#2309) * Perform force repair when polaris not found (apache#2313) * fix(deps): update dependency com.github.spotbugs:spotbugs-annotations to v4.9.4 (apache#2312) * chore(deps): update dependency pre-commit to v4.3.0 (apache#2311) * fix(deps): update dependency software.amazon.awssdk:bom to v2.32.19 (apache#2314) * fix(deps): update dependency com.gradleup.shadow:shadow-gradle-plugin to v9.0.1 (apache#2310) * Separate Cloud Integration Tests (apache#2283) Currently, cloud integration tests are part of the regular test suite but require cloud credentials to execute properly. This creates unnecessary overhead for developers who just want to run local builds or for CI jobs that don't have cloud access configured. * chore(deps): update actions/checkout action to v5 (apache#2319) * CatalogEntity: internal endpoint not considered (apache#2292) Setting an S3 internal endpoint doesn't work, because the property's not carried over from the OpenAPI model type into `AwsStorageConfigInfo`. * client/python: loosen boto3 dep (apache#2188) It's hard to depend on this package when the boto3 dependency is so strict. * fix(deps): update dependency io.micrometer:micrometer-bom to v1.15.3 (apache#2321) * NoSQL: test adjustment * Last merged commit 45e8e02 --------- Co-authored-by: Alexandre Dutra <adutra@apache.org> Co-authored-by: Yong Zheng <yongzheng0809@gmail.com> Co-authored-by: Mend Renovate <bot@renovateapp.com> Co-authored-by: Christopher Lambert <xn137@gmx.de> Co-authored-by: Yujiang Zhong <42907416+zhongyujiang@users.noreply.github.com> Co-authored-by: Pooja Nilangekar <poojan@umd.edu> Co-authored-by: Tamas Mate <50709850+tmater@users.noreply.github.com> Co-authored-by: Tyler Rockwood <rockwotj@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR addresses the gap in Hadoop federation to ensure that we initialize the hadoop configuration prior to initializing the catalog object.
The iceberg library expects the hadoop configuration to be initialized before creating a HadoopCatalog object. This change ensures that Polaris is compatible with the underlying iceberg library. Additionally, since the config initialization is based on the underlying (default) core-site.xml file, the change ensures that the federated catalog was created using
IMPLICITauthentication mode.Testing:
Due to current limitations in the current test setup, tested the change manually.
[TODO] Add a regtest with hadoop federation once the change is baked into the apache/polaris docker image.