Skip to content

Commit da546bd

Browse files
snazyadutrarenovate-botXN137poojanilangekar
authored
Dremio merge 2025 08 18 11 02 (apache#104)
* Fix Keycloak getting-started example (apache#2349) The `polaris-setup` container was erroneously including a non-existent scope when fetching a token from Keycloak. * fix(deps): update dependency com.nimbusds:nimbus-jose-jwt to v10.4.2 (apache#2350) * Use PolarisTaskConstants (apache#2346) * Add a regression test for Catalog Federation (apache#2286) * Add a regression test for Catalog Federation * Install jq dependency * Fix token issues * Update regtests/README.md Co-authored-by: Eric Maynard <emaynard@apache.org> * Update README.md --------- Co-authored-by: Eric Maynard <emaynard@apache.org> * Modularize federation (Option 2) (apache#2332) * Modularize federation (Option 2) * Move polaris-extensions-federation-hadoop dependency * Change identifier to lowerCase * Change identifiers to constants * Replace CallContext with RealmConfig in enforceFeatureEnabledOrThrow (apache#2348) * Replace CallContext with RealmConfig in CatalogEntity (apache#2336) * chore(deps): update registry.access.redhat.com/ubi9/openjdk-21-runtime docker tag to v1.23-6 (apache#2353) * fix(deps): update dependency com.gradleup.shadow:shadow-gradle-plugin to v9.0.2 (apache#2358) * chore(deps): update postgres docker tag to v17.6 (apache#2354) * Add integration tests with Keycloak (apache#2343) * Fix REST responses for failed Admin operations (apache#2291) * Fix REST responses for failed Admin operations the `boolean` return values of many methods in `PolarisAdminService` were often simply not getting used at all, thus the REST api returned success in those cases even though the `PrivilegeResult` was marked as failed. due to this fix a silently failing test now needs to be adjusted. we return the `PrivilegeResult` instead of a `boolean` to give the client at least some indication of what has gone wrong on the server side. note that some of the other operations were throwing Expcetions already, which are already reported back correctly to the client. * review: use http 400 BAD_REQUEST * Make PolarisAuthorizer RequestScoped (apache#2340) all methods in `PolarisAuthorizer` currently have a `CallContext` parameter. in its only implementation only `CallContext.getRealmConfig` is getting used. so since `PolarisAuthorizer` cant be used outside a request, we can simply make it request-scoped and inject the request-scoped `RealmConfig` directly. * fix(deps): update mockito monorepo to v5.19.0 (apache#2360) * Fix soft-merge conflict on `main` (apache#2364) * feat(docs): Add Getting Stated guide for MinIO (apache#2227) * feat(docs): Add Getting Stated guide for MinIO A simple page of step-by-step instructions for setting up a local environment with Polaris, MinIO and Spark. Closes apache#1530 * IntelliJ: fix project icon in IJ project list (apache#2366) ... copy source has changed * Use asMap property helpers (apache#2347) seems like these helpers existed for a long time but were just not getting used consistently * SigV4 Auth Support for Catalog Federation - Part 2: Connection Config Persistence (apache#2190) * Add SigV4 related DPOs * Rename UserSecretReference to SecretReference and fix some small issues * fix(deps): update dependency software.amazon.awssdk:bom to v2.32.24 (apache#2371) * Rat-check: exclude venv, cleanup excludes, include .svg (apache#2363) * `.svg` files are XML files and can contain a license header * Re-grouped the exclusion rat patterns * Added exclude for `.venv` * Added exclude for `.ruff_cache` * NoSQL: Async-impls: add some safeguards + javadoc spelling * NoSQL: spelling * NoSQL: dependency updates * Last merged commit: 5a7686b --------- Co-authored-by: Alexandre Dutra <adutra@apache.org> Co-authored-by: Mend Renovate <bot@renovateapp.com> Co-authored-by: Christopher Lambert <xn137@gmx.de> Co-authored-by: Pooja Nilangekar <nilangekar.pooja@gmail.com> Co-authored-by: Eric Maynard <emaynard@apache.org> Co-authored-by: Dmitri Bourlatchkov <dmitri.bourlatchkov@gmail.com> Co-authored-by: Rulin Xing <xjdkcsq3@gmail.com>
1 parent 058bc4f commit da546bd

File tree

198 files changed

+3580
-1430
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

198 files changed

+3580
-1430
lines changed

‎build.gradle.kts‎

Lines changed: 36 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@ if (System.getProperty("idea.sync.active").toBoolean()) {
5050
val icon = ideaDir.file("icon.png").asFile
5151
if (!icon.exists()) {
5252
copy {
53-
from("docs/img/logos/polaris-brandmark.png")
53+
from("site/static/img/logos/polaris-brandmark.png")
5454
into(ideaDir)
5555
rename { _ -> "icon.png" }
5656
}
@@ -60,11 +60,15 @@ if (System.getProperty("idea.sync.active").toBoolean()) {
6060
eclipse { project { name = ideName } }
6161

6262
tasks.named<RatTask>("rat").configure {
63-
// These are Gradle file pattern syntax
63+
// Gradle
6464
excludes.add("**/build/**")
65+
excludes.add("gradle/wrapper/gradle-wrapper*")
66+
excludes.add(".gradle")
67+
excludes.add("**/kotlin-compiler*")
68+
excludes.add("**/build-logic/.kotlin/**")
6569

66-
excludes.add("docs/CNAME")
67-
excludes.add("docs/index.html")
70+
excludes.add("ide-name.txt")
71+
excludes.add("version.txt")
6872

6973
excludes.add("DISCLAIMER_WIP")
7074
excludes.add("LICENSE")
@@ -85,64 +89,61 @@ tasks.named<RatTask>("rat").configure {
8589
// Manifest files do not allow comments
8690
excludes.add("tools/version/src/jarTest/resources/META-INF/FAKE_MANIFEST.MF")
8791

88-
excludes.add(
89-
"persistence/nosql/persistence/index/src/testFixtures/resources/org/apache/polaris/persistence/indexes/words.gz"
90-
)
91-
92-
excludes.add("ide-name.txt")
93-
excludes.add("version.txt")
92+
// Git & GitHub
9493
excludes.add(".git")
95-
excludes.add(".gradle")
96-
excludes.add(".idea")
97-
excludes.add(".java-version")
98-
excludes.add("**/.keep")
99-
excludes.add("**/poetry.lock")
100-
10194
excludes.add(".github/pull_request_template.md")
10295

103-
excludes.add("spec/docs.yaml")
104-
excludes.add("spec/index.yml")
96+
// Misc build artifacts
97+
excludes.add(".java-version")
98+
excludes.add("**/.keep")
99+
excludes.add("logs/**")
100+
excludes.add("**/*.lock")
105101

106-
excludes.add("gradle/wrapper/gradle-wrapper*")
102+
// Binary files
103+
excludes.add(
104+
"persistence/nosql/persistence/index/src/testFixtures/resources/org/apache/polaris/persistence/indexes/words.gz"
105+
)
107106

108-
excludes.add("logs/**")
107+
// Polaris service startup banner
109108
excludes.add("runtime/service/src/**/banner.txt")
110109

110+
// Web site
111+
excludes.add("**/go.sum")
111112
excludes.add("site/node_modules/**")
112113
excludes.add("site/layouts/robots.txt")
113114
// Ignore generated stuff, when the Hugo is run w/o Docker
114115
excludes.add("site/public/**")
115116
excludes.add("site/resources/_gen/**")
116117
excludes.add("node_modules/**")
117118

119+
// Python
120+
excludes.add("**/.venv/**")
118121
excludes.add("**/polaris-venv/**")
119-
122+
excludes.add("**/poetry.lock")
123+
excludes.add("**/.ruff_cache/**")
124+
excludes.add("**/.mypy_cache/**")
120125
excludes.add("**/.pytest_cache/**")
126+
excludes.add("client/python/.openapi-generator/**")
127+
128+
// Jupyter
129+
excludes.add("**/*.ipynb")
130+
131+
// regtests
121132
excludes.add("regtests/**/py.typed")
122133
excludes.add("regtests/**/*.ref")
123134
excludes.add("regtests/.env")
124135
excludes.add("regtests/derby.log")
125136
excludes.add("regtests/metastore_db/**")
126-
excludes.add("client/python/.openapi-generator/**")
127137
excludes.add("regtests/output/**")
138+
excludes.add("plugins/**/*.ref")
128139

129-
excludes.add("**/*.ipynb")
140+
// IntelliJ
141+
excludes.add(".idea")
130142
excludes.add("**/*.iml")
131143
excludes.add("**/*.iws")
132144

145+
// Rat can't scan binary images
133146
excludes.add("**/*.png")
134-
excludes.add("**/*.svg")
135-
136-
excludes.add("**/*.lock")
137-
138-
excludes.add("**/*.env*")
139-
140-
excludes.add("**/go.sum")
141-
142-
excludes.add("**/kotlin-compiler*")
143-
excludes.add("**/build-logic/.kotlin/**")
144-
145-
excludes.add("plugins/**/*.ref")
146147
}
147148

148149
tasks.register<Exec>("regeneratePythonClient") {

‎extensions/federation/hadoop/build.gradle.kts‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one
3+
* or more contributor license agreements. See the NOTICE file
4+
* distributed with this work for additional information
5+
* regarding copyright ownership. The ASF licenses this file
6+
* to you under the Apache License, Version 2.0 (the
7+
* "License"); you may not use this file except in compliance
8+
* with the License. You may obtain a copy of the License at
9+
*
10+
* http://www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing,
13+
* software distributed under the License is distributed on an
14+
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
* KIND, either express or implied. See the License for the
16+
* specific language governing permissions and limitations
17+
* under the License.
18+
*/
19+
20+
plugins {
21+
id("polaris-client")
22+
alias(libs.plugins.jandex)
23+
}
24+
25+
dependencies {
26+
// Polaris dependencies
27+
implementation(project(":polaris-core"))
28+
29+
implementation(platform(libs.iceberg.bom))
30+
implementation("org.apache.iceberg:iceberg-api")
31+
implementation("org.apache.iceberg:iceberg-core")
32+
implementation("org.apache.iceberg:iceberg-common")
33+
34+
// Hadoop dependencies (for Hadoop catalog support)
35+
implementation(libs.hadoop.common) {
36+
exclude("org.slf4j", "slf4j-reload4j")
37+
exclude("org.slf4j", "slf4j-log4j12")
38+
exclude("ch.qos.reload4j", "reload4j")
39+
exclude("log4j", "log4j")
40+
exclude("org.apache.zookeeper", "zookeeper")
41+
exclude("org.apache.hadoop.thirdparty", "hadoop-shaded-protobuf_3_25")
42+
exclude("com.github.pjfanning", "jersey-json")
43+
exclude("com.sun.jersey", "jersey-core")
44+
exclude("com.sun.jersey", "jersey-server")
45+
exclude("com.sun.jersey", "jersey-servlet")
46+
exclude("io.dropwizard.metrics", "metrics-core")
47+
}
48+
implementation(libs.hadoop.client.api)
49+
implementation(libs.hadoop.client.runtime)
50+
51+
// CDI dependencies for runtime discovery
52+
implementation(libs.jakarta.enterprise.cdi.api)
53+
implementation(libs.smallrye.common.annotation)
54+
55+
// Logging
56+
implementation(libs.slf4j.api)
57+
}

‎extensions/federation/hadoop/src/main/java/org/apache/polaris/extensions/federation/hadoop/HadoopFederatedCatalogFactory.java‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one
3+
* or more contributor license agreements. See the NOTICE file
4+
* distributed with this work for additional information
5+
* regarding copyright ownership. The ASF licenses this file
6+
* to you under the Apache License, Version 2.0 (the
7+
* "License"); you may not use this file except in compliance
8+
* with the License. You may obtain a copy of the License at
9+
*
10+
* http://www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing,
13+
* software distributed under the License is distributed on an
14+
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
* KIND, either express or implied. See the License for the
16+
* specific language governing permissions and limitations
17+
* under the License.
18+
*/
19+
package org.apache.polaris.extensions.federation.hadoop;
20+
21+
import io.smallrye.common.annotation.Identifier;
22+
import jakarta.enterprise.context.ApplicationScoped;
23+
import org.apache.hadoop.conf.Configuration;
24+
import org.apache.iceberg.catalog.Catalog;
25+
import org.apache.iceberg.hadoop.HadoopCatalog;
26+
import org.apache.polaris.core.catalog.ExternalCatalogFactory;
27+
import org.apache.polaris.core.connection.AuthenticationParametersDpo;
28+
import org.apache.polaris.core.connection.AuthenticationType;
29+
import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
30+
import org.apache.polaris.core.connection.ConnectionType;
31+
import org.apache.polaris.core.connection.hadoop.HadoopConnectionConfigInfoDpo;
32+
import org.apache.polaris.core.secrets.UserSecretsManager;
33+
import org.slf4j.Logger;
34+
import org.slf4j.LoggerFactory;
35+
36+
/** Factory class for creating a Hadoop catalog handle based on connection configuration. */
37+
@ApplicationScoped
38+
@Identifier(ConnectionType.HADOOP_FACTORY_IDENTIFIER)
39+
public class HadoopFederatedCatalogFactory implements ExternalCatalogFactory {
40+
private static final Logger LOGGER = LoggerFactory.getLogger(HadoopFederatedCatalogFactory.class);
41+
42+
@Override
43+
public Catalog createCatalog(
44+
ConnectionConfigInfoDpo connectionConfigInfoDpo, UserSecretsManager userSecretsManager) {
45+
// Currently, Polaris supports Hadoop federation only via IMPLICIT authentication.
46+
// Hence, prior to initializing the configuration, ensure that the catalog uses
47+
// IMPLICIT authentication.
48+
AuthenticationParametersDpo authenticationParametersDpo =
49+
connectionConfigInfoDpo.getAuthenticationParameters();
50+
if (authenticationParametersDpo.getAuthenticationTypeCode()
51+
!= AuthenticationType.IMPLICIT.getCode()) {
52+
throw new IllegalStateException("Hadoop federation only supports IMPLICIT authentication.");
53+
}
54+
Configuration conf = new Configuration();
55+
String warehouse = ((HadoopConnectionConfigInfoDpo) connectionConfigInfoDpo).getWarehouse();
56+
HadoopCatalog hadoopCatalog = new HadoopCatalog(conf, warehouse);
57+
hadoopCatalog.initialize(
58+
warehouse, connectionConfigInfoDpo.asIcebergCatalogProperties(userSecretsManager));
59+
return hadoopCatalog;
60+
}
61+
}

‎getting-started/assets/polaris/create-catalog.sh‎

Lines changed: 13 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -23,23 +23,16 @@ apk add --no-cache jq
2323

2424
realm=${1:-"POLARIS"}
2525

26-
token=${2:-""}
26+
TOKEN=${2:-""}
2727

28-
if [ -z "$token" ]; then
29-
token=$(curl -s http://polaris:8181/api/catalog/v1/oauth/tokens \
30-
--user ${CLIENT_ID}:${CLIENT_SECRET} \
31-
-H "Polaris-Realm: $realm" \
32-
-d grant_type=client_credentials \
33-
-d scope=PRINCIPAL_ROLE:ALL | jq -r .access_token)
28+
BASEDIR=$(dirname $0)
3429

35-
if [ -z "${token}" ]; then
36-
echo "Failed to obtain access token."
37-
exit 1
38-
fi
30+
if [ -z "$TOKEN" ]; then
31+
source $BASEDIR/obtain-token.sh
3932
fi
4033

4134
echo
42-
echo "Obtained access token: ${token}"
35+
echo "Obtained access token: ${TOKEN}"
4336

4437
STORAGE_TYPE="FILE"
4538
if [ -z "${STORAGE_LOCATION}" ]; then
@@ -57,12 +50,14 @@ else
5750
echo "Using StorageType: $STORAGE_TYPE"
5851
fi
5952

60-
STORAGE_CONFIG_INFO="{\"storageType\": \"$STORAGE_TYPE\", \"allowedLocations\": [\"$STORAGE_LOCATION\"]}"
53+
if [ -z "${STORAGE_CONFIG_INFO}" ]; then
54+
STORAGE_CONFIG_INFO="{\"storageType\": \"$STORAGE_TYPE\", \"allowedLocations\": [\"$STORAGE_LOCATION\"]}"
6155

62-
if [[ "$STORAGE_TYPE" == "S3" ]]; then
63-
STORAGE_CONFIG_INFO=$(echo "$STORAGE_CONFIG_INFO" | jq --arg roleArn "$AWS_ROLE_ARN" '. + {roleArn: $roleArn}')
64-
elif [[ "$STORAGE_TYPE" == "AZURE" ]]; then
65-
STORAGE_CONFIG_INFO=$(echo "$STORAGE_CONFIG_INFO" | jq --arg tenantId "$AZURE_TENANT_ID" '. + {tenantId: $tenantId}')
56+
if [[ "$STORAGE_TYPE" == "S3" ]]; then
57+
STORAGE_CONFIG_INFO=$(echo "$STORAGE_CONFIG_INFO" | jq --arg roleArn "$AWS_ROLE_ARN" '. + {roleArn: $roleArn}')
58+
elif [[ "$STORAGE_TYPE" == "AZURE" ]]; then
59+
STORAGE_CONFIG_INFO=$(echo "$STORAGE_CONFIG_INFO" | jq --arg tenantId "$AZURE_TENANT_ID" '. + {tenantId: $tenantId}')
60+
fi
6661
fi
6762

6863
echo
@@ -82,7 +77,7 @@ PAYLOAD='{
8277

8378
echo $PAYLOAD
8479

85-
curl -s -H "Authorization: Bearer ${token}" \
80+
curl -s -H "Authorization: Bearer ${TOKEN}" \
8681
-H 'Accept: application/json' \
8782
-H 'Content-Type: application/json' \
8883
-H "Polaris-Realm: $realm" \

‎getting-started/assets/polaris/obtain-token.sh‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
#
2+
# Licensed to the Apache Software Foundation (ASF) under one
3+
# or more contributor license agreements. See the NOTICE file
4+
# distributed with this work for additional information
5+
# regarding copyright ownership. The ASF licenses this file
6+
# to you under the Apache License, Version 2.0 (the
7+
# "License"); you may not use this file except in compliance
8+
# with the License. You may obtain a copy of the License at
9+
#
10+
# http://www.apache.org/licenses/LICENSE-2.0
11+
#
12+
# Unless required by applicable law or agreed to in writing,
13+
# software distributed under the License is distributed on an
14+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
# KIND, either express or implied. See the License for the
16+
# specific language governing permissions and limitations
17+
# under the License.
18+
#
19+
20+
set -e
21+
22+
apk add --no-cache jq
23+
24+
realm=${1:-"POLARIS"}
25+
26+
TOKEN=$(curl -s http://polaris:8181/api/catalog/v1/oauth/tokens \
27+
--user ${CLIENT_ID}:${CLIENT_SECRET} \
28+
-H "Polaris-Realm: $realm" \
29+
-d grant_type=client_credentials \
30+
-d scope=PRINCIPAL_ROLE:ALL | jq -r .access_token)
31+
32+
if [ -z "${TOKEN}" ]; then
33+
echo "Failed to obtain access token."
34+
exit 1
35+
fi
36+
37+
export TOKEN

‎getting-started/assets/postgres/docker-compose-postgres.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@
1919

2020
services:
2121
postgres:
22-
image: postgres:17.5
22+
image: postgres:17.6
2323
ports:
2424
- "5432:5432"
2525
# set shared memory limit when using docker-compose

‎getting-started/keycloak/docker-compose.yml‎

Lines changed: 10 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -67,13 +67,16 @@ services:
6767
- CLIENT_SECRET=s3cr3t
6868
volumes:
6969
- ../assets/polaris/:/polaris
70-
entrypoint: |
71-
/bin/sh -c "apk add --no-cache jq && \
72-
chmod +x /polaris/create-catalog.sh && \
73-
token=$$(curl http://keycloak:8080/realms/iceberg/protocol/openid-connect/token --user client1:s3cr3t -d 'grant_type=client_credentials' -d 'scope=catalog' | jq -r .access_token) && \
74-
/polaris/create-catalog.sh realm-internal && \
75-
/polaris/create-catalog.sh realm-external $$token && \
76-
/polaris/create-catalog.sh realm-mixed $$token"
70+
entrypoint: "/bin/sh"
71+
command:
72+
- "-c"
73+
- >-
74+
apk add --no-cache jq &&
75+
chmod +x /polaris/create-catalog.sh &&
76+
token=$$(curl http://keycloak:8080/realms/iceberg/protocol/openid-connect/token --user client1:s3cr3t -d 'grant_type=client_credentials' | jq -r .access_token) &&
77+
/polaris/create-catalog.sh realm-internal &&
78+
/polaris/create-catalog.sh realm-external $$token &&
79+
/polaris/create-catalog.sh realm-mixed $$token
7780
7881
keycloak:
7982
image: quay.io/keycloak/keycloak:26.3.2

0 commit comments

Comments
 (0)