Dremio merge 2025 05 27 08 46 (#76)

* fix(nightly-CI): Do not publish snapshots from forks (apache#1635)

Adopt the `Nightly Build` workflow to not (try to) publish every night from forks.

* main: Update dependency io.smallrye.config:smallrye-config-core to v3.13.0 (apache#1637)

* Use echo to print script errors (apache#1648)

* [HOTFIX] QUICKSTART (apache#1646)

The change adds the following to fix Quick start experience : 
[1] ENV variables required by common assets after apache#1522 
[2] New configs required to enable FILE based sources apache#1649

Co-authored-by: singhpk234 <singhpk234@users.noreply.github.com>
Co-authored-by: pjanuario <pjanuario@users.noreply.github.com>

* main: Update dependency gradle to v8.14.1 (apache#1652)

* main: Update dependency gradle to v8.14.1

* Re-adopt PR to the project's needs

---------

Co-authored-by: Robert Stupp <snazy@snazy.de>

* [Policy Store] Add policyTypeCode to Slice/Index for Future Filtering Support and Update Policy Persistence Method (apache#1628)

This PR adds policyTypeCode to the in-memory tree map store's slice and the SQL index on policy_mapping_records (already done in JDBC in apache#1468). This prepares for future features that need to filter efficiently by policy type, like listing all entities with a data compaction policy.

It also updates the loadAllTargetsOnPolicy method to accept policyTypeCode, enabling it to use the new index for better performance.

* fix(test): Do not let some more tests spam `/tmp` (apache#1651)

* fix(test): Do not let some more tests not spam `/tmp`

* `PolarisRestCatalogViewFileIntegrationTest`
* `FileIOExceptionsTest`
* `PolarisRestCatalogViewFileIntegrationTest`

Changes the tests to leverage JUnit's `@TempDir`.

Simplifies `PolarisEclipseLinkMetaStoreManagerTest`

* review: rename the (now) abstract class

* fix(testing): Do not let PolarisOverlappingTableTest spam `/tmp` (apache#1641)

Changes the test to leverage JUnit's `@TempDir`.

* Add CATALOG_MANAGE_METADATA to super privilege set of policy attachment privileges (apache#1643)

* Fix quickstart doc with docker compose (apache#1610)

* main: Update dependency boto3 to v1.38.22 (apache#1657)

* Refactor IcebergCatalog to isolate internal state (apache#1659)

Following up on apache#1694

* Restore `private` scope on internal fields in `IcebergCatalog`

* Use a test-only setter instead of sub-classing to manage injecting
  test FileIO implementations

* Refactor: Use per-request STS credentials (apache#1629)

* Refactor: Use per-request STS credentials

No functional changes.

This is mostly to allow more storage integration
flexibility in downstream build.

This might also be useful for non-AWS storage.

* fix and enforce more errorprone checks (apache#1663)

enforces the following checks:
https://errorprone.info/bugpattern/ObjectsHashCodePrimitive
https://errorprone.info/bugpattern/OptionalMapToOptional
https://errorprone.info/bugpattern/StringCharset
https://errorprone.info/bugpattern/VariableNameSameAsType

* Create a wrapper script to generate python client; regenerate the python client (apache#1347)

As noted in apache#755 and elsewhere, the generated types in client/python are currently out of date. This introduces a script to regenerate them and a gradle task to run that script.

I've also run the script, which necessitated several things to get tests passing:

1. There were small nonfunctional spec changes needed in order to keep the Python client working
2. The CLI and its tests required a few fixes to work with the updated Python client
3. Many of the regtests required fixes to work with the updated Python client

* [Python Client] CI for Python client (Continue PR#1096) (apache#1639)

Adds CI for python client. It does not include caching poetry step for now since we do not have poetry.lock (it is in .gitignore), see relevant discussion in: apache#1102 (comment), apache#1096 (comment), we can add that later

* main: Update actions/setup-python action to v5 (apache#1671)

* main: Update actions/checkout action to v4 (apache#1670)

* main: Update python Docker tag to v3.13 (apache#1669)

* main: Update dependency pytest to ~=7.4.4 (apache#1668)

* main: Update dependency software.amazon.awssdk:bom to v2.31.50 (apache#1677)

* main: Update dependency boto3 to v1.38.23 (apache#1667)

* feat(build): make archive builds reproducible (apache#1664)

See https://docs.gradle.org/current/userguide/working_with_files.html#sec:reproducible_archives

* main: Update dependency io.prometheus:prometheus-metrics-exporter-servlet-jakarta to v1.3.8 (apache#1679)

* NoSQL: adapt to change on oss/main

* INFO: Last merged commit: 6ef8b3e

---------

Co-authored-by: Mend Renovate <bot@renovateapp.com>
Co-authored-by: ModEtchFill <50123102+ModEtchFill@users.noreply.github.com>
Co-authored-by: Prashant Singh <35593236+singhpk234@users.noreply.github.com>
Co-authored-by: singhpk234 <singhpk234@users.noreply.github.com>
Co-authored-by: pjanuario <pjanuario@users.noreply.github.com>
Co-authored-by: Honah (Jonas) J. <honahx@apache.org>
Co-authored-by: MonkeyCanCode <yongzheng0809@gmail.com>
Co-authored-by: Dmitri Bourlatchkov <dmitri.bourlatchkov@dremio.com>
Co-authored-by: Dmitri Bourlatchkov <dmitri.bourlatchkov@gmail.com>
Co-authored-by: Christopher Lambert <xn137@gmx.de>
Co-authored-by: Eric Maynard <eric.maynard+oss@snowflake.com>

Author: 10 people

.github/workflows/nightly.yml

@@ -43,6 +43,7 @@ permissions:
 jobs:
   nightly_build:
     runs-on: ubuntu-latest
+    if: github.repository == 'apache/polaris'
     steps:
       - name: Checkout
         uses: actions/checkout@v4

.github/workflows/python-client.yml

@@ -0,0 +1,70 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+# This workflow uses actions that are not certified by GitHub.
+# They are provided by a third-party and are governed by
+# separate terms of service, privacy policy, and support
+# documentation.
+# This workflow will build a Python project with Poetry and cache/restore any dependencies to improve the workflow execution time
+# For more information see: https://docs.github.com/en/actions/use-cases-and-examples/building-and-testing/building-and-testing-python
+
+name: Python Client CI
+
+on:
+  push:
+    branches: [ "main" ]
+  pull_request:
+    branches: [ "main" ]
+
+jobs:
+  build:
+
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.9", "3.10", "3.11"]
+
+    steps:
+      - name: Checkout Polaris project
+        uses: actions/checkout@v4
+
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - name: Install Poetry
+        run: |
+          pip install --user --upgrade -r regtests/requirements.txt
+
+      # TODO: add cache for poetry dependencies once we have poetry.lock in the repo
+      - name: Install dependencies
+        working-directory: client/python
+        run: poetry install --all-extras
+
+      - name: Lint with flake8
+        working-directory: client/python
+        run: |
+          poetry run flake8 --select=E9,F63,F7,F82 --max-line-length=127 --statistics
+
+      - name: Test with pytest
+        working-directory: client/python
+        run: |
+          export SCRIPT_DIR="non-existing-mock-directory"
+          poetry run pytest

api/iceberg-service/src/main/java/org/apache/polaris/service/types/NotificationType.java

@@ -19,7 +19,6 @@
 package org.apache.polaris.service.types;
 
 import java.util.Arrays;
-import java.util.Locale;
 import java.util.Map;
 import java.util.Optional;
 import java.util.stream.Collectors;
@@ -84,9 +83,9 @@ public static Optional<NotificationType> lookupByName(String name) {
       return Optional.empty();
     }
 
-    for (NotificationType NotificationType : NotificationType.values()) {
-      if (name.toUpperCase(Locale.ROOT).equals(NotificationType.name())) {
-        return Optional.of(NotificationType);
+    for (NotificationType notificationType : NotificationType.values()) {
+      if (name.equalsIgnoreCase(notificationType.name())) {
+        return Optional.of(notificationType);
       }
     }
     return Optional.empty();

build-logic/src/main/kotlin/polaris-java.gradle.kts

@@ -213,3 +213,10 @@ configurations.all {
       }
     }
 }
+
+// ensure jars conform to reproducible builds
+// (https://docs.gradle.org/current/userguide/working_with_files.html#sec:reproducible_archives)
+tasks.withType<AbstractArchiveTask>().configureEach {
+  isPreserveFileTimestamps = false
+  isReproducibleFileOrder = true
+}

build.gradle.kts

@@ -145,6 +145,18 @@ tasks.named<RatTask>("rat").configure {
   excludes.add("plugins/**/*.ref")
 }
 
+tasks.register<Exec>("regeneratePythonClient") {
+  description = "Regenerates the python client"
+
+  workingDir = project.projectDir
+  commandLine("bash", "client/templates/regenerate.sh")
+
+  dependsOn(":polaris-api-iceberg-service:processResources")
+  dependsOn(":polaris-api-management-service:processResources")
+  dependsOn(":polaris-api-catalog-service:processResources")
+  dependsOn(":polaris-api-management-model:processResources")
+}
+
 // Pass environment variables:
 //    ORG_GRADLE_PROJECT_apacheUsername
 //    ORG_GRADLE_PROJECT_apachePassword

client/python/.github/workflows/python.yml

@@ -22,7 +22,7 @@
 #
 # ref: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python
 
-name: polaris.management Python package
+name: polaris.catalog Python package
 
 on: [push, pull_request]
 

client/python/.gitlab-ci.yml

@@ -0,0 +1,50 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+# NOTE: This file is auto generated by OpenAPI Generator.
+# URL: https://openapi-generator.tech
+#
+# ref: https://docs.gitlab.com/ee/ci/README.html
+# ref: https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Python.gitlab-ci.yml
+
+stages:
+  - test
+
+.pytest:
+  stage: test
+  script:
+   - pip install -r requirements.txt
+   - pip install -r test-requirements.txt
+   - pytest --cov=polaris.catalog
+
+pytest-3.7:
+  extends: .pytest
+  image: python:3.13-alpine
+pytest-3.8:
+  extends: .pytest
+  image: python:3.13-alpine
+pytest-3.9:
+  extends: .pytest
+  image: python:3.13-alpine
+pytest-3.10:
+  extends: .pytest
+  image: python:3.13-alpine
+pytest-3.11:
+  extends: .pytest
+  image: python:3.13-alpine