Dremio merge 2025 09 17 18 51 (apache#119)

* Avoid exceptions on ETag matches (apache#2578)

Exceptions have runtime overhead, which is avoidable in this case.

* Publish build scans to develocity.apache.org for build insights (apache#2559)

* Site: add a blog for apache doris and polaris integration (apache#2571)

* Bump: iceberg 1.10 (apache#2586)

* Python client: add license check (apache#2580)

* Python client: add license check

* Python client: add license check

* Python client: add license check

* test

* Enable license check

* Fix license check for cryptography (apache#2591)

* Update plugin com.gradle.common-custom-user-data-gradle-plugin to v2.4.0 (apache#2582)

* Update plugin com.gradle.develocity to v4.2 (apache#2583)

* Build: remove unnecessary openapigenerator plugin usages (apache#2592)

* docs(changelog): Update the Changelog with 1.0.1, 1.0.0, & 0.9.0 Releases (apache#2587)

* Revert "Update plugin com.gradle.develocity to v4.2 (apache#2583)" (apache#2594)

This reverts commit 8cc0fad, build scans are not published to the ASF Develocity instance.

* Add security report page on the website (apache#2538)

This closes apache#2521

* CI: Fix Gradle cache usages (apache#2593)

The GH action `ci-incr-build-cache-prepare` calls the `setup-gradle` action with the "right" parameters, so the call of `setup-gradle` in jobs that use `ci-incr-build-cache-prepare` is superfluous. But this also renders the caching ineffective as `setup-gradle` is called w/o `cache-read-only: true` in "child jobs", which then attempt to store their Gradle cache contents. This leads to attempty storing the Gradle cache in `Post Setup Gradle` steps of the "child" jobs, conflicting with other child jobs and also conflicting with `Store Gradle Cache` purpose.

"Which cache" is then restored, is rather non-deterministic, but almost always at least "partial" leading to unnecessary (re)builds.

* Last merged commit f3d53f0

---------

Co-authored-by: Dmitri Bourlatchkov <dmitri.bourlatchkov@gmail.com>
Co-authored-by: Clay Johnson <cjohnson@gradle.com>
Co-authored-by: Mingyu Chen (Rayner) <yunyou@selectdb.com>
Co-authored-by: Prashant Singh <35593236+singhpk234@users.noreply.github.com>
Co-authored-by: Yong Zheng <yongzheng0809@gmail.com>
Co-authored-by: Mend Renovate <bot@renovateapp.com>
Co-authored-by: Adam Christian <105929021+adam-christian-software@users.noreply.github.com>
Co-authored-by: JB Onofré <jbonofre@apache.org>

Author: 9 people

.github/workflows/gradle.yml

@@ -46,16 +46,13 @@ jobs:
         with:
           java-version: '21'
           distribution: 'temurin'
-      - name: Setup Gradle
-        uses: gradle/actions/setup-gradle@ed408507eac070d1f99cc633dbcf757c94c7933a # v4
-        with:
-          validate-wrappers: false
       - name: Prepare Gradle build cache
         uses: ./.github/actions/ci-incr-build-cache-prepare
       - name: Run unit tests
         env:
           # publishToMavenLocal causes a GH API requests, use the token for those requests
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}
         run: |
           ./gradlew check sourceTarball distTar distZip publishToMavenLocal \
             -x :polaris-runtime-service:test \
@@ -87,13 +84,11 @@ jobs:
         with:
           java-version: '21'
           distribution: 'temurin'
-      - name: Setup Gradle
-        uses: gradle/actions/setup-gradle@ed408507eac070d1f99cc633dbcf757c94c7933a # v4
-        with:
-          validate-wrappers: false
       - name: Prepare Gradle build cache
         uses: ./.github/actions/ci-incr-build-cache-prepare
       - name: Run Quarkus tests
+        env:
+          DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}
         run: |
           ./gradlew \
             :polaris-runtime-service:test \
@@ -125,14 +120,12 @@ jobs:
         with:
           java-version: '21'
           distribution: 'temurin'
-      - name: Setup Gradle
-        uses: gradle/actions/setup-gradle@ed408507eac070d1f99cc633dbcf757c94c7933a # v4
-        with:
-          validate-wrappers: false
       - name: Prepare Gradle build cache
         uses: ./.github/actions/ci-incr-build-cache-prepare
       - name: Run integration tests
         run: ./gradlew intTest --continue
+        env:
+          DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}
       - name: Save partial Gradle build cache
         uses: ./.github/actions/ci-incr-build-cache-save
         if: github.event_name == 'push' && github.ref == 'refs/heads/main'
@@ -163,14 +156,13 @@ jobs:
         with:
           java-version: '21'
           distribution: 'temurin'
-      - name: Setup Gradle
-        uses: gradle/actions/setup-gradle@ed408507eac070d1f99cc633dbcf757c94c7933a # v4
-        with:
-          validate-wrappers: false
       - name: Collect partial Gradle build caches
         uses: ./.github/actions/ci-incr-build-cache-prepare
         with:
           cache-read-only: false
       - name: Trigger Gradle home cleanup
+        env:
+          DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}
         run: ./gradlew --no-daemon :showVersion
+
       # Note: the "Post Gradle invocation" archives the updated build cache.

.github/workflows/helm.yml

@@ -83,6 +83,8 @@ jobs:
           minikube docker-env
 
       - name: Image build
+        env:
+          DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}
         run: |
           eval $(minikube -p minikube docker-env)
           ./gradlew \

.github/workflows/nightly.yml

@@ -52,14 +52,14 @@ jobs:
         with:
           java-version: '21'
           distribution: 'temurin'
-      - name: Setup Gradle
-        uses: gradle/actions/setup-gradle@ed408507eac070d1f99cc633dbcf757c94c7933a # v4
-        with:
-          validate-wrappers: false
+
+      - name: Prepare Gradle build cache
+        uses: ./.github/actions/ci-incr-build-cache-prepare
       - name: Publish SNAPSHOTs to Apache Nexus Repository
         run: ./gradlew publishToApache
         env:
           # publishToApache causes a GH API requests, use the token for those requests
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           ORG_GRADLE_PROJECT_apacheUsername: ${{ secrets.NEXUS_USER }}
-          ORG_GRADLE_PROJECT_apachePassword: ${{ secrets.NEXUS_PW }} 
+          ORG_GRADLE_PROJECT_apachePassword: ${{ secrets.NEXUS_PW }}
+          DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}

.github/workflows/python-client.yml

@@ -59,11 +59,17 @@ jobs:
         run: |
           make client-lint
 
+      - name: License Compliance Check
+        run: |
+          make client-license-check
+
       - name: Generated Client Tests
         run: |
           make client-unit-test
 
       - name: Image build
+        env:
+          DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}
         run: |
           ./gradlew \
             :polaris-server:assemble \

.github/workflows/regtest.yml

@@ -40,18 +40,15 @@ jobs:
           java-version: '21'
           distribution: 'temurin'
 
-      - name: Setup Gradle
-        uses: gradle/actions/setup-gradle@ed408507eac070d1f99cc633dbcf757c94c7933a # v4
-        with:
-          validate-wrappers: false
-
       - name: Prepare Gradle build cache
         uses: ./.github/actions/ci-incr-build-cache-prepare
 
       - name: Fix permissions
         run: mkdir -p regtests/output && chmod 777 regtests/output && chmod 777 regtests/t_*/ref/*
 
       - name: Image build
+        env:
+          DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}
         run: |
           ./gradlew \
               :polaris-server:assemble \

.github/workflows/spark_client_regtests.yml

@@ -40,11 +40,6 @@ jobs:
           java-version: '21'
           distribution: 'temurin'
 
-      - name: Setup Gradle
-        uses: gradle/actions/setup-gradle@ed408507eac070d1f99cc633dbcf757c94c7933a # v4
-        with:
-          validate-wrappers: false
-
       - name: Prepare Gradle build cache
         uses: ./.github/actions/ci-incr-build-cache-prepare
 
@@ -55,9 +50,12 @@ jobs:
         env:
           # publishToMavenLocal causes a GH API requests, use the token for those requests
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}
         run: ./gradlew assemble publishToMavenLocal
 
       - name: Image build
+        env:
+          DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}
         run: |
           ./gradlew \
               :polaris-server:assemble \

CHANGELOG.md

@@ -41,9 +41,9 @@ request adding CHANGELOG notes for breaking (!) changes and possibly other secti
 
 - **MinIO**: Added MinIO integration support with comprehensive getting started documentation.
 
-### Upgrade notes
+### Upgrade Notes
 
-### Breaking changes
+### Breaking Changes
 
 - Helm chart: the default value of the `authentication.tokenBroker.secret.symmetricKey.secretKey` property has changed
   from `symmetric.pem` to `symmetric.key`.
@@ -105,9 +105,45 @@ refresh-credentials flag for the desired storage provider.
 
 ### Commits
 
-## [1.0.0-incubating]
+## [1.0.1-incubating]
+Apache Polaris 1.0.1-incubating was released on August 16th, 2025. It’s a maintenance release on the 1.0.0 release fixing a couple of issues on the Helm Chart:
+- remove db-kind in Helm Chart
+- add relational-jdbc to helm
+
 
-- TODO: backfill 1.0.0 release notes
+## [1.0.0-incubating]
+Apache Polaris 1.0.0-incubating was released on July 9th, 2025.
+
+- **Highlights**
+  - First release ready for real-world workloads after the public beta 0.9.0 
+  - **Binary distribution** – first release with single downloadable .tgz or .zip artifact. 
+  - **Helm Chart** – debut of an official Helm chart for seamless Kubernetes deployment 
+- **New features & enhancements**
+  - **Policy Store** — persistence with schema evolution, built‑in TMS policies (Data compaction, Snapshot expiry, etc) and REST CRUD endpoints 
+  - **Postgres JDBC persistence** — native JDBC backend with robust support for concurrent changes. 
+  - **Rollback Compaction on Conflicts** - makes Polaris smarter, to revert the compaction commits in case of crunch to let the writers who are actually adding or removing the data to the table succeed. In a sense treating compaction as always a lower priority process. 
+  - **Enhanced runtime** — new runtime powered by Quarkus delivers out‑of‑the‑box Kubernetes readiness, quick startup, OIDC integration, and many other benefits. Polaris server and admin tool are now using Quarkus as a runtime framework. 
+  - **HTTP caching via ETag** — the loadTable endpoint supports ETag, reducing bandwidth and improving perceived latency 
+  - **Support for external identity providers (IdP)** — Polaris can now be its own IdP, delegate to an external IdP, or both 
+  - **Snapshot filtering** – clients can choose to load only referenced snapshots 
+  - **Catalog Federation (experimental)** – federate requests to an external Iceberg REST or Hadoop Catalog 
+  - **Generic Tables (experimental)** — serve multiple table formats besides Iceberg tables; initial Spark 3.5 plugin supports Delta Lake 
+  - **Event Listener framework (experimental)** — subscribe to catalog events (AfterTableCommitedEvent, BeforeViewCommitedEvent, etc)
+- **Notable bug fixes**
+  - **CLI and Python Client improvements** – Support for new features, CLI repair, changes to the update subcommand, and various fixes 
+  - **Safe configurations** – Catalog-level Polaris configurations follow a strict naming convention to avoid name clashes with user-provided configuration entries. Legacy Polaris configuration names are still supported in 1.0 to allow existing deployments to migrate without rush. 
+  - **TableOperations optimizations** – Changes to BasePolarisTableOperations result in less traffic to object storage during commits 
+  - **Bounded entity cache** – The entity cache is now more memory-aware and less likely to lead to OOMs 
+  - **Bootstrapping fixes** – Users can more easily bootstrap a new realm. Root credentials can be provided by the user or generated by Polaris (and returned to the user). 
+- **Breaking changes**
+  - **Server Configuration** – The format used to configure the Polaris service in 0.9 has changed with the migration to Quarkus and changes to configurations 
+  - **Bootstrap Flow** – The bootstrap flow used in 0.9 has changed with the migration to Quarkus and the new admin tool
+
+
+## [0.9.0-incubating]
+Apache Polaris 0.9.0 was released on March 11, 2025 as the first Polaris release. Only the source distribution is available for this release.
 
 [Unreleased]: https://github.com/apache/polaris/commits
+[1.0.1-incubating]: https://github.com/apache/polaris/releases/tag/apache-polaris-1.0.1-incubating
 [1.0.0-incubating]: https://github.com/apache/polaris/releases/tag/apache-polaris-1.0.0-incubating-rc2
+[0.9.0-incubating]: https://github.com/apache/polaris/releases/tag/apache-polaris-0.9.0-incubating

Makefile

@@ -162,6 +162,12 @@ client-integration-test: client-setup-env ## Run client integration tests
 	@echo "Tearing down Docker Compose services..."
 	@$(DOCKER) compose -f $(PYTHON_CLIENT_DIR)/docker-compose.yml down || true # Ensure teardown even if tests fail
 
+.PHONY: client-license-check
+client-license-check: client-setup-env ## Run license compliance check
+	@echo "--- Starting license compliance check ---"
+	@$(ACTIVATE_AND_CD) && pip-licenses
+	@echo "--- License compliance check complete ---"
+
 .PHONY: client-build
 client-build: client-setup-env ## Build client distribution
 	@echo "--- Building client distribution ---"

README.md

@@ -136,6 +136,19 @@ Default configuration values can be found in `runtime/defaults/src/main/resource
   ```
 - See [README in `site/`](site/README.md) for more information.
 
+#### Publishing Build Scans to develocity.apache.org
+
+All authenticated builds of Apache Polaris will automatically publish build scans to the ASF Develocity instance at 
+[develocity.apache.org](https://develocity.apache.org/scans?search.rootProjectNames=polaris). 
+
+CI builds originating from the `apache/polaris` repository will have access to the Apache organization-level secret 
+`DEVELOCITY_ACCESS_KEY` and publish build scans using the secret. CI builds originating from pull requests from forks 
+will not have access to the secret and will silently skip build scan publication.
+
+Apache committers can publish build scans from their local machine by 
+[provisioning an access key](https://docs.gradle.com/develocity/gradle-plugin/current/#automated_access_key_provisioning) 
+using ASF LDAP credentials. Builds by anonymous, unauthenticated contributors will silently skip build scan publication.
+
 ## License
 
 Apache Polaris is under the Apache License Version 2.0. See the [LICENSE](LICENSE).

client/python/pyproject.toml

@@ -62,10 +62,40 @@ mypy = ">=1.18, <=1.18.1"
 pyiceberg = "==0.10.0"
 pre-commit = "==4.3.0"
 openapi-generator-cli = "==7.11.0.post0"
+pip-licenses = "==5.0.0"
 # pin virtualenv version to prevent poetry from upgrading to an incompatible version
 # see https://github.com/python-poetry/poetry/issues/10504#issuecomment-3176923981
 virtualenv = ">=20.26.6,<20.33.0"
 
+[tool.pip-licenses]
+from-classifier = true
+# Packages with "UNKNOWN" licenses in pip-licenses metadata.
+# These have been manually verified and are known to be compatible with ASF.
+ignore-packages = [
+    "anyio",                 # MIT License (MIT)
+    "build",                 # MIT License (MIT)
+    "CacheControl",          # Apache-2.0
+    "cffi",                  # MIT License (MIT)
+    "click",                 # BSD-3-Clause
+    "cryptography",          # Apache-2.0 or BSD-3-Clause
+    "fsspec",                # BSD-3-Clause
+    "jaraco.functools",      # MIT License (MIT)
+    "jeepney",               # MIT License (MIT)
+    "more-itertools",        # MIT License (MIT)
+    "mypy_extensions",       # MIT License (MIT)
+    "pyparsing",             # MIT License (MIT)
+    "RapidFuzz",             # MIT License (MIT)
+    "SecretStorage",         # BSD-3-Clause
+    "types-python-dateutil", # Apache-2.0
+    "typing-inspection",     # MIT License (MIT)
+    "typing_extensions",     # PSF-2.0
+    "urllib3",               # MIT License (MIT)
+    "zipp",                  # MIT License (MIT)
+    "zstandard",             # BSD-3-Clause
+]
+partial-match = true
+allow-only = "MIT;Apache;BSD License;PSF-2.0;ISC;The Unlicense;Python Software Foundation License;Mozilla Public License"
+
 [build-system]
 requires = ["poetry-core>=2.0.0,<3.0.0", "openapi-generator-cli==7.11.0.post0"]
 build-backend = "poetry.core.masonry.api"