Dremio merge 2025 06 02 10 57 (#78)

* main: Update dependency org.postgresql:postgresql to v42.7.6 (apache#1697)

* main: Update helm/chart-testing-action action to v2.7.0 (apache#1700)

* main: Update gradle/actions digest to 8379f6a (apache#1696)

* main: Update medyagh/setup-minikube action to v0.0.19 (apache#1698)

* feat(metrics): Mitigate potential performance issues with realm_id tag (apache#1662)

As discussed in the ML, this PR introduces two flags to enable the inclusion of realm ID tags in API and HTTP metrics.

They are both disabled by default.

There is also a new safeguard: if the cardinality of realm IDs in HTTP metrics goes above a configurable threshold (100 by default), a warning is printed and no more HTTP metrics will be recorded. (Quarkus has a similar safeguard for URI tags in HTTP metrics.)

* Site/contributing: add recommendations for working with PRs (apache#1625)

This change updates the PR guidelines on the "Contributing" web page after [this discussion](https://lists.apache.org/thread/kfxo3cqmw3pgrpgtgqvqpwvn46yw8q7h).

Also adopt `gradlew test` to `gradlew check` in README, following the intent (all tests, incl ITs)

* main: Update dependency com.azure:azure-sdk-bom to v1.2.35 (apache#1703)

* main: Update dependency com.adobe.testing:s3mock-testcontainers to v4.4.0 (apache#1705)

* main: Update dependency boto3 to v1.38.24 (apache#1702)

* Keep generated RSA-key-pair for JWT token broker on heap (apache#1661)

Polaris allows using RSA key-paris for the JWT token broker. The recommended way is to [generate the RSA key pair](https://github.com/apache/polaris/blob/d8b862b13914d526ee147dc0e359bfc9c1e319ad/site/content/in-dev/unreleased/configuring-polaris-for-production.md?plain=1#L61-L66) and configure the location of the key files.

However, if only `polaris.authentication.token-broker.type=rsa-key-pair` but not the `public/private-key-pair` options are configured, Polaris generates those and stores them in `/tmp` using random file names (using `Files.createTempFile()`) - this happens for each (matching) realm. Each Polaris startup generates new key-pairs for each of those realms. It's practically not possible to associate the files to a realm. There is already a [production readiness check](https://github.com/apache/polaris/blob/d8b862b13914d526ee147dc0e359bfc9c1e319ad/quarkus/service/src/main/java/org/apache/polaris/service/quarkus/config/ProductionReadinessChecks.java#L118-L166) to warn users about this behavior.

Due to the issue that the files cannot be associated, those seem to be somewhat useless and bring no advantage over keeping these "ephemeral RSA key pairs" on heap. This PR changes the code to not write the key-pair to the file system and keeps these "ephemeral key pairs" on heap. Since the same code path is used for key-paris _provided_ by the user (via the `public/private-key-pair` config options), that code path now only reads those files once and not every time the private/public key is needed.

* Merge JPA module with EclipseLink Module (apache#1718)

* Create LICENSE and NOTICE for "single" distribution (apache#1694)

* Fix a failing task with the release profile (apache#1693)

* Remove unused adminDocs artifact (apache#1749)

* Production readiness for Persistence (apache#1707)

Production readiness for Persistence (apache#1707)

* Fixes for direct usage of client_secret apache#1756

When the spec was upgraded and the python client regenerated from it, clientSecret was made a password, which means calling str on it directly yields a redacted string like ******. In the initial PR to change the python client and fix regtests, some existing usage of client_secret was not changed.

* main: Update dependency org.junit:junit-bom to v5.13.0 (apache#1760)

* main: Update dependency org.testcontainers:testcontainers-bom to v1.21.1 (apache#1748)

* fix: Improve reliability of metrics tests (apache#1763)

CI sometimes fails with errors like "http_server_requests_seconds not found"
in the reported metrics.

This looks like a race between the Quarkus metrics producer and the
tests asking for these metrics.

This change adds a time-limited retry loop until the expected metrics
are available, before proceeding with other assertions.

Note: in normal cases the loop finishes fast because the metrics are
available. The two-minute timeout would apply only when the expected
metrics fail to be produced at all.

* Fix test_spark_credentials_s3_exception_on_metadata_file_deletion (apache#1759)

* Regenerate bundled spec & Regenerate Python client (apache#1751)

I ran these commands from main:

```
redocly bundle spec/polaris-catalog-service.yaml -o spec/generated/bundled-polaris-catalog-service.yaml
./gradlew regeneratePythonClient
```

I didn't realize before that some Python types are generated form the bundled spec, so some of the fixes from apache#1347 didn't get properly applied before.

* main: Update dependency boto3 to v1.38.27 (apache#1714)

* NoSQL: bump Weld/Junit5 (fixes a bug that surfaces w/ JUnit 5.13)

* NoSQL: Let some more tests leverage Jandex

* Info: Last merged commit b7aac72

---------

Co-authored-by: Mend Renovate <bot@renovateapp.com>
Co-authored-by: Alexandre Dutra <adutra@users.noreply.github.com>
Co-authored-by: Yufei Gu <yufei@apache.org>
Co-authored-by: JB Onofré <jbonofre@apache.org>
Co-authored-by: Prashant Singh <35593236+singhpk234@users.noreply.github.com>
Co-authored-by: Eric Maynard <eric.maynard+oss@snowflake.com>
Co-authored-by: Dmitri Bourlatchkov <dmitri.bourlatchkov@gmail.com>
Co-authored-by: gh-yzou <167037035+gh-yzou@users.noreply.github.com>

Author: 9 people

.github/workflows/gradle.yml

@@ -50,7 +50,7 @@ jobs:
       # Configure Gradle for optimal use in GiHub Actions, including caching of downloaded dependencies.
       # See: https://github.com/gradle/actions/blob/main/setup-gradle/README.md
       - name: Setup Gradle
-        uses: gradle/actions/setup-gradle@06832c7b30a0129d7fb559bcc6e43d26f6374244 # v4
+        uses: gradle/actions/setup-gradle@8379f6a1328ee0e06e2bb424dadb7b159856a326 # v4
         with:
           # The setup-gradle action fails, if the wrapper is not using the right version or is not present.
           # Our `gradlew` validates the integrity of the `gradle-wrapper.jar`, so it's safe to disable this.

.github/workflows/helm.yml

@@ -55,7 +55,7 @@ jobs:
           version: 'v3.16.0'
 
       - name: Set up chart-testing
-        uses: helm/chart-testing-action@v2.6.1
+        uses: helm/chart-testing-action@0d28d3144d3a25ea2cc349d6e59901c4ff469b3b # v2.7.0
 
       - name: Run 'helm template' validation
         run: |
@@ -75,7 +75,7 @@ jobs:
         run: ct lint --target-branch ${{ github.event.repository.default_branch }} --debug --charts ./helm/polaris
 
       - name: Set up Minikube
-        uses: medyagh/setup-minikube@v0.0.18
+        uses: medyagh/setup-minikube@cea33675329b799adccc9526aa5daccc26cd5052 # v0.0.19
 
       - name: Print Docker info
         run: |

CONTRIBUTING.md

@@ -64,32 +64,60 @@ Before starting on a large code change, please describe the concept/design of wh
 
 The best way to provide changes is to fork Apache Polaris repository on GitHub and provide a Pull Request with your changes. To make it easy to apply your changes please use the following conventions:
 
-* Every Pull Request should have a matching GitHub Issue.
-* Create a branch that will house your change:
-
-```bash
-git clone https://github.com/apache/polaris
-cd polaris
-git fetch --all
-git checkout -b my-branch origin/main
-```
-
-  Don't forget to periodically rebase your branch:
-
-```bash
-git pull --rebase
-git push GitHubUser my-branch --force
-```
-
-  Ensure the code is properly formatted:
-
-```bash
-./gradlew format
-```
+#### Before opening a pull request
 
 * Pull Requests should be based on the `main` branch.
-* Test that your changes works by adapting or adding tests. Verify the build passes (see `README.md` for build instructions).
+* Create a branch that will house your change:
+  ```bash
+  git clone https://github.com/apache/polaris
+  cd polaris
+  git checkout main
+  git pull
+  git checkout -b my-branch
+  ```
+* Work on the changes of your pull requests locally.
+* Recommended checks:
+  ```bash
+  # Ensure the code is properly formatted and compiles:
+  ./gradlew format compileAll
+  # Ensure the code is passing the checks (including formatting checks & tests):
+  ./gradlew check
+  ```
+* You may want to push your changes to your personal Polaris fork. Git will emit a URL that you can use to create the Pull Request. Do not create the Pull Request yet.
+  ```bash
+  git push --set-upstream your-github-accout
+  ```
+
+#### Opening a Pull Request
+
+* The Pull Request summary should provide a concise summary of the change, get inspired by [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/).
+* The Pull Request description should provide the background (rationale) of the change and describe the changes in way that someone who has no prior knowledge can understand the rationale of the change and the change itself.
+* If there is a matching GitHub Issue, add a separate line at the end of the commit message of the issue that the PR fixes. Do not add the issue number into the subject.
+  ```
+  Fixes #123456
+  ```
+  If the PR does not fully fix the issue, use `Related to #123456` instead of `Fixes #123456`.
+
+Tips:
+* If the branch for your Pull Request contains only one (squashed) commit, GitHub will populate the PR summary and description from that single commit.
+* When opening a PR consider whether the PR is "draft" or already "ready for review". "Draft" means work in progress, things will change, but comments are welcome. "Ready for review" means that the PR is requested to be merged as is (pending review feedback).
+
+#### Working on a Pull Request
+
+* Don't forget to periodically rebase your branch:
+  ```bash
+  git pull --rebase
+  git push your-github-accout my-branch --force
+  ```
+* Test that your changes work by adapting or adding tests. Verify the build passes (see `README.md` for build instructions).
 * If your Pull Request has conflicts with the `main` branch, please rebase and fix the conflicts.
+* If your PR requires more work or time or bigger changes, please put the PR to "draft" state to indicate that it is not meant to be "thoroughly" reviewed at this point.
+
+#### Merging a Pull Request
+
+* When a PR is about to be merged, cross-check the commit summary and message for the merged Git commit.
+* Keep in mind that the Git commit subject and message is going to be read by other people, potentially even after years. The Git commit subject and message will appear "as is" in release notes.
+* Make sure the subject and message are properly formatted and contains a concise description of the changes in way that someone who has no prior knowledge can understand the rationale of the change and the change itself. Remove information that's of no use for someone reading the Git commit log, for example single intermediate commit messages like `formatting` or `fix test`.
 
 ## Java version requirements
 

README.md

@@ -59,15 +59,14 @@ Apache Polaris is organized into the following modules:
   - `polaris-quarkus-server` - The Polaris server runtime
   - `polaris-quarkus-admin-tool` - The Polaris admin & maintenance tool
 - Persistence modules:
-  - `polaris-jpa-model` - The JPA entity definitions
   - `polaris-eclipselink` - The Eclipselink implementation of the MetaStoreManager interface
   - `polaris-relational-jdbc` - The JDBC implementation of BasePersistence to be used via AtomicMetaStoreManager
 
 Apache Polaris is built using Gradle with Java 21+ and Docker 27+.
 
 - `./gradlew build` - To build and run tests. Make sure Docker is running, as the integration tests depend on it.
 - `./gradlew assemble` - To skip tests.
-- `./gradlew test` - To run unit tests and integration tests.
+- `./gradlew check` - To run all checks, including unit tests and integration tests.
 - `./gradlew run` - To run the Polaris server locally; the server is reachable at localhost:8181. This is also suitable for running regression tests, or for connecting with Spark. Set your own credentials by specifying system property `./gradlew run -Dpolaris.bootstrap.credentials=POLARIS,root,secret` where:
   - `POLARIS` is the realm
   - `root` is the CLIENT_ID

bom/build.gradle.kts

@@ -86,7 +86,6 @@ dependencies {
 
     api(project(":polaris-eclipselink"))
     api(project(":polaris-relational-jdbc"))
-    api(project(":polaris-jpa-model"))
 
     api(project(":polaris-quarkus-admin"))
     api(project(":polaris-quarkus-common"))

client/python/docs/AuthenticationParameters.md

@@ -20,7 +20,7 @@
 -->
 # AuthenticationParameters
 
-Authentication-specific information for a REST connection
+Authentication-specific information for a connection
 
 ## Properties
 

client/python/docs/AwsIamServiceIdentityInfo.md

@@ -0,0 +1,49 @@
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements.  See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership.  The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License.  You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied.  See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+# AwsIamServiceIdentityInfo
+
+
+## Properties
+
+Name | Type | Description | Notes
+------------ | ------------- | ------------- | -------------
+**iam_arn** | **str** | The ARN of the IAM user or IAM role Polaris uses to assume roles and then access external resources. | 
+
+## Example
+
+```python
+from polaris.management.models.aws_iam_service_identity_info import AwsIamServiceIdentityInfo
+
+# TODO update the JSON string below
+json = "{}"
+# create an instance of AwsIamServiceIdentityInfo from a JSON string
+aws_iam_service_identity_info_instance = AwsIamServiceIdentityInfo.from_json(json)
+# print the JSON string representation of the object
+print(AwsIamServiceIdentityInfo.to_json())
+
+# convert the object into a dict
+aws_iam_service_identity_info_dict = aws_iam_service_identity_info_instance.to_dict()
+# create an instance of AwsIamServiceIdentityInfo from a dict
+aws_iam_service_identity_info_from_dict = AwsIamServiceIdentityInfo.from_dict(aws_iam_service_identity_info_dict)
+```
+[[Back to Model list]](../README.md#documentation-for-models) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to README]](../README.md)
+
+

client/python/docs/CatalogPrivilege.md

@@ -73,6 +73,22 @@
 
 * `VIEW_FULL_METADATA` (value: `'VIEW_FULL_METADATA'`)
 
+* `POLICY_CREATE` (value: `'POLICY_CREATE'`)
+
+* `POLICY_WRITE` (value: `'POLICY_WRITE'`)
+
+* `POLICY_READ` (value: `'POLICY_READ'`)
+
+* `POLICY_DROP` (value: `'POLICY_DROP'`)
+
+* `POLICY_LIST` (value: `'POLICY_LIST'`)
+
+* `POLICY_FULL_METADATA` (value: `'POLICY_FULL_METADATA'`)
+
+* `CATALOG_ATTACH_POLICY` (value: `'CATALOG_ATTACH_POLICY'`)
+
+* `CATALOG_DETACH_POLICY` (value: `'CATALOG_DETACH_POLICY'`)
+
 [[Back to Model list]](../README.md#documentation-for-models) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to README]](../README.md)
 
 

client/python/docs/ConnectionConfigInfo.md

@@ -29,6 +29,7 @@ Name | Type | Description | Notes
 **connection_type** | **str** | The type of remote catalog service represented by this connection | 
 **uri** | **str** | URI to the remote catalog service | [optional] 
 **authentication_parameters** | [**AuthenticationParameters**](AuthenticationParameters.md) |  | [optional] 
+**service_identity** | [**ServiceIdentityInfo**](ServiceIdentityInfo.md) |  | [optional] 
 
 ## Example
 

client/python/docs/HadoopConnectionConfigInfo.md

@@ -0,0 +1,50 @@
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements.  See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership.  The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License.  You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied.  See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+# HadoopConnectionConfigInfo
+
+Configuration necessary for connecting to a Hadoop Catalog
+
+## Properties
+
+Name | Type | Description | Notes
+------------ | ------------- | ------------- | -------------
+**warehouse** | **str** | The file path to where this catalog should store tables | [optional] 
+
+## Example
+
+```python
+from polaris.management.models.hadoop_connection_config_info import HadoopConnectionConfigInfo
+
+# TODO update the JSON string below
+json = "{}"
+# create an instance of HadoopConnectionConfigInfo from a JSON string
+hadoop_connection_config_info_instance = HadoopConnectionConfigInfo.from_json(json)
+# print the JSON string representation of the object
+print(HadoopConnectionConfigInfo.to_json())
+
+# convert the object into a dict
+hadoop_connection_config_info_dict = hadoop_connection_config_info_instance.to_dict()
+# create an instance of HadoopConnectionConfigInfo from a dict
+hadoop_connection_config_info_from_dict = HadoopConnectionConfigInfo.from_dict(hadoop_connection_config_info_dict)
+```
+[[Back to Model list]](../README.md#documentation-for-models) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to README]](../README.md)
+
+