Merge OSS/main, adopt to changes from Generalize bootstrapping in servers (#53)

* Policy Store: PolicyMappingRecord with Persistence Impl (apache#1104)

* Spark: Setup repository code structure and build (apache#1190)

* Added freshness aware table loading using metadata file location for ETag (apache#1037)

* Pulled in iceberg 1.8.0 spec changes for freshness aware table loading and added feature to Polaris

* Changed etag support to use entityId:version tuple

* fixed getresponse call

* Changed etagged response to record and gave default implementation to ETaggableEntity

* Made iceberg rest spec docs clearer

* Added HTTP Compliant ETag and IfNoneMatch representations and separated persistence from etag logic

* Changed ETag to be a record and improved semantics of IfNoneMatch

* Fixed semantics of if none match

* Removed ETag representation, consolidated in IfNoneMatch

* fixed if none match parsing

* Added table entity retrieval method to table operations

* removed accidental commit of pycache folders

* Fixed formatting

* Changed to use metadata location hash

* Ran formatting

* use sha256

* Moved out ETag functions to utility class and removed ETaggedLoadTableResponse

* Addressed comments

* Fixed IcebergTableLikeEntity package rename

* main: Update dependency io.opentelemetry.semconv:opentelemetry-semconv to v1.31.0 (apache#1288)

* Update LICENSE and NOTICE in the distributions (admin and server) (apache#1258)

* Gradle/Quarkus: make imageBuild task depend on jandex (apache#1290)

* Core: Clarify the atomicity of BasePersistence methods (apache#1274)

* Implement GenericTableCatalogAdapter (apache#1264)

* rebase

* more fixes

* autolint

* working on tests

* stable test

* autolint

* polish

* changes per review

* some changes per review

* grants

* autolint

* changes per review

* changes per review

* typofix

* Improve code-containment and efficiency of etag-aware loading (apache#1296)

* Improve code-containment and efficiency of etag-aware loading

-Make the hash generation resilient against null metadataLocation
-Use getResolvedPath instead of getPassthroughResolvedPath to avoid redundant persistence round-trip
-Only try to calculate the etag for comparison against ifNoneMatch if the ifNoneMatch is actually provided

* Add strict null-checking at callsites to generateETag, disallow passing null to generator

* Add TODO to refactor shared logic for etag generation

* Core: Add Endpoints and resource paths for Generic Table (apache#1286)

* main: Update dependency com.nimbusds:nimbus-jose-jwt to v10.1 (apache#1299)

* [JDBC] Part1 : ADD SQL script for Polaris setup (apache#1276)

* main: Update registry.access.redhat.com/ubi9/openjdk-21-runtime Docker tag to v1.22-1.1743605859 (apache#1300)

* done (apache#1297)

* Add Polaris Community Meeting for April 3, 2025 (apache#1304)

* Use config-file to define errorprone rule (apache#1233)

Also enabled a couple more simple rules, and adding suppressions/fixes for/to the code.

The two rules `EqualsGetClass` and `UnusedMethod`, which I think are useful, are not enabled yet, because that would mean actual code changes, which I do not want to do in this PR.

The rule `PatternMatchingInstanceof`, introduced in apache#393, is disabled in this PR. It does not work before errorrpone 2.37.0 (via apache#1213) - requires additional changes to enable the rule (see apache#1215).

* Add Yun as a contributor (apache#1310)

* Refactor CatalogHandler to comply with ErrorProne rules (apache#1312)

Fix the CI error after apache#1233

* Implement PolicyCatalog Stage 1: CRUD + ListPolicies (apache#1294)

* main: Update dependency io.opentelemetry:opentelemetry-bom to v1.49.0 (apache#1316)

* main: Update docker.io/jaegertracing/all-in-one Docker tag to v1.68.0 (apache#1317)

* main: Update dependency boto3 to v1.37.28 (apache#1328)

* main: Update dependency software.amazon.awssdk:bom to v2.31.16 (apache#1329)

* Make `BasePolaritsMetaStoreManagerTest` and `(Base)ResolverTest` reusable (apache#1308)

Moves the test cases into the `Base*` classes and make sure the classes can be reused by other persistence implementations.

* main: Update dependency io.opentelemetry.semconv:opentelemetry-semconv to v1.32.0 (apache#1293)

* main: Update mockito monorepo to v5.17.0 (apache#1311)

* PySpark Update AWS Region (apache#1302)

Co-authored-by: Travis Bowen <travis.bowen@snowflake.com>

* main: Update dependency com.nimbusds:nimbus-jose-jwt to v10.2 (apache#1334)

* main: Update dependency com.diffplug.spotless:spotless-plugin-gradle to v7.0.3 (apache#1335)

* Maven publication: Produce correct `<scm><tag>` in `pom.xml` (apache#1330)

`project.scm.tag` in a Maven pom is intended to refer to the SCM (Git) tag. We currently publish `main`, which is incorrect.

This change omits the SCM tag for snapshot builds, but emits the Git tag for releases.

* Remove `@StaticInitSafe` annotation (apache#1331)

There was an issue around mapped configurations having the `@StaticInitSafe` annotation that led to _two_ instances (a "static" one and a "somewhet application-scoped" one) - this was fixed in Quarkus 3.21. One bug in smallrye-config is fixed for Quarkus > 3.21.0, another issue however remains.

Since `@StaticInitSafe` annotated configs seem to cause some weird issues, it seems legit to remote that annotation altogether. This approach was [taken in Nessie](projectnessie/nessie#10606) as well. Investigations (via practical experiments) have proven that there's no measurable impact (runtime + heap) when doing this - and that's also been confirmed by Quarkus + Smallrye-config maintainers.

Hence this change remotes that annotation from the code base.

* Build/Release: Add a "generate digest" task and use for source tarball and Quarkus distributables (apache#1271)

* Ensure that digest and signature are generated for both Polaris-Server and admin tar/zip distribution
* Move "generate digest" functionality to a Gradle task

* main: Update dependency com.google.errorprone:error_prone_core to v2.37.0 (apache#1213)

* main: Update Quarkus Platform and Group to v3.21.1 (apache#1291)

* main: Update dependency io.netty:netty-codec-http2 to v4.2.0.Final (apache#1301)

* Remove unnecessary `clean` and `--no-build-cache` from Gradle invocations (apache#1338)

`quarkusAppPartsBuild --rerun` is the right way to force a Docker image build.

* Generalize bootstrapping in servers (apache#1313)

* Remove `instanceof` checks from `QuarkusProducers`.

* Remove the now unused `onStartup` method from `InMemoryPolarisMetaStoreManagerFactory`.

* Instead, call the good old `bootstrapRealms` method from `QuarkusProducers`.

* Add new config property to control which MetaStore types are bootstrapped automatically (defaults to `in-memory` as before).

* There is no bootstrap behaviour change in this PR, only refactorings to simplify code.

* Add info log message to indicate when a realm is bootstrapped in runtime using preset credentials.

Future enhancements may include pulling preset credentials from a secret manager like Vault for bootstrapping (s discussed in comments on apache#1228).

* main: Update actions/stale digest to 816d9db (apache#1341)

* main: Update dependency com.adobe.testing:s3mock-testcontainers to v4 (apache#1342)

* Revert "Introduce MetaStoreManagerFactory.initializeForService"

This reverts commit 23abc6e322de66b8aadccdf6b2614cf8ab658ffe.

* NoSQL: Adopt to change in MSMF

---------

Co-authored-by: Honah (Jonas) J. <honahx@apache.org>
Co-authored-by: gh-yzou <167037035+gh-yzou@users.noreply.github.com>
Co-authored-by: Mansehaj Singh <msehajs@gmail.com>
Co-authored-by: Mend Renovate <bot@renovateapp.com>
Co-authored-by: JB Onofré <jbonofre@apache.org>
Co-authored-by: Alexandre Dutra <adutra@users.noreply.github.com>
Co-authored-by: Yufei Gu <yufei@apache.org>
Co-authored-by: Eric Maynard <eric.maynard+oss@snowflake.com>
Co-authored-by: Dennis Huo <7410123+dennishuo@users.noreply.github.com>
Co-authored-by: Prashant Singh <35593236+singhpk234@users.noreply.github.com>
Co-authored-by: Travis Bowen <122238243+travis-bowen@users.noreply.github.com>
Co-authored-by: Travis Bowen <travis.bowen@snowflake.com>
Co-authored-by: Dmitri Bourlatchkov <dmitri.bourlatchkov@dremio.com>

Author: 14 people

.github/workflows/helm.yml

@@ -99,7 +99,11 @@ jobs:
         if: steps.list-changed.outputs.changed == 'true'
         run: |
           eval $(minikube -p minikube docker-env)
-          ./gradlew :polaris-quarkus-server:assemble :polaris-quarkus-admin:assemble \
+          ./gradlew \
+            :polaris-quarkus-server:assemble \
+            :polaris-quarkus-server:quarkusAppPartsBuild --rerun \
+            :polaris-quarkus-admin:assemble \
+            :polaris-quarkus-admin:quarkusAppPartsBuild --rerun \
             -Dquarkus.container-image.build=true \
             -PeclipseLinkDeps=org.postgresql:postgresql:42.7.4
           minikube image ls

.github/workflows/regtest.yml

@@ -45,7 +45,10 @@ jobs:
 
       - name: Image build
         run: |
-          ./gradlew :polaris-quarkus-server:assemble -Dquarkus.container-image.build=true
+          ./gradlew \
+              :polaris-quarkus-server:assemble \
+              :polaris-quarkus-server:quarkusAppPartsBuild --rerun \
+              -Dquarkus.container-image.build=true
 
       - name: Regression Test
         env:

.github/workflows/stale.yml

@@ -22,7 +22,7 @@ jobs:
   stale:
     runs-on: ubuntu-24.04
     steps:
-      - uses: actions/stale@ba23c1cb02e5cb8f885b0994d870e6032be00186
+      - uses: actions/stale@816d9db1aba399a7f70277f1a2b01a4d21497fdd
         with:
           days-before-close: 5
           days-before-stale: 30

README.md

@@ -76,8 +76,13 @@ select * from db1.table1;
 
 #### Running in Docker
 
-- `./gradlew clean :polaris-quarkus-server:assemble -Dquarkus.container-image.build=true --no-build-cache` - To 
-  build the image locally.
+- To build the image locally:
+  ```bash
+  ./gradlew \
+    :polaris-quarkus-server:assemble \
+    :polaris-quarkus-server:quarkusAppPartsBuild --rerun \
+    -Dquarkus.container-image.build=true
+  ```
 - `docker run -p 8181:8181 -p 8182:8182 apache/polaris:latest` - To run the image.
 
 The Polaris codebase contains some docker compose examples to quickly get started with Polaris,

extension/persistence/eclipselink/src/main/java/org/apache/polaris/extension/persistence/impl/eclipselink/EclipseLinkPolarisMetaStoreManagerFactory.java

@@ -24,7 +24,6 @@
 import jakarta.enterprise.context.ApplicationScoped;
 import jakarta.inject.Inject;
 import java.nio.file.Path;
-import java.util.List;
 import org.apache.polaris.core.PolarisDiagnostics;
 import org.apache.polaris.core.context.RealmContext;
 import org.apache.polaris.core.persistence.LocalPolarisMetaStoreManagerFactory;
@@ -55,10 +54,6 @@ protected EclipseLinkPolarisMetaStoreManagerFactory(PolarisDiagnostics diagnosti
     super(diagnostics);
   }
 
-  @Override
-  public void initializeForService(
-      @Nonnull List<String> realmIds, @Nonnull String defaultRealmId) {}
-
   @Override
   protected PolarisEclipseLinkStore createBackingStore(@Nonnull PolarisDiagnostics diagnostics) {
     return new PolarisEclipseLinkStore(diagnostics);

getting-started/eclipselink/README.md

@@ -25,11 +25,14 @@ This example requires `jq` to be installed on your machine.
    the Postgres JDBC driver:
 
     ```shell
-    ./gradlew clean :polaris-quarkus-server:assemble :polaris-quarkus-admin:assemble \
+    ./gradlew \
+       :polaris-quarkus-server:assemble \
+       :polaris-quarkus-server:quarkusAppPartsBuild --rerun \
+       :polaris-quarkus-admin:assemble \
+       :polaris-quarkus-admin:quarkusAppPartsBuild --rerun \
        -PeclipseLinkDeps=org.postgresql:postgresql:42.7.4 \
        -Dquarkus.container-image.tag=postgres-latest \
-       -Dquarkus.container-image.build=true \
-       --no-build-cache
+       -Dquarkus.container-image.build=true
     ```
 
 2. Start the docker compose group by running the following command from the root of the repository:

getting-started/spark/README.md

@@ -27,7 +27,10 @@ A Jupyter notebook is used to run PySpark.
 If a Polaris image is not already present locally, build one with the following command:
 
 ```shell
-./gradlew clean :polaris-quarkus-server:assemble -Dquarkus.container-image.build=true --no-build-cache
+./gradlew \
+  :polaris-quarkus-server:assemble \
+  :polaris-quarkus-server:quarkusAppPartsBuild --rerun \
+  -Dquarkus.container-image.build=true
 ```
 
 ## Run the `docker-compose` file

getting-started/telemetry/README.md

@@ -24,7 +24,10 @@ This example requires `jq` to be installed on your machine.
 1. Build the Polaris image if it's not already present locally:
 
     ```shell
-    ./gradlew clean :polaris-quarkus-server:assemble -Dquarkus.container-image.build=true --no-build-cache
+    ./gradlew \
+      :polaris-quarkus-server:assemble \
+      :polaris-quarkus-server:quarkusAppPartsBuild --rerun \
+      -Dquarkus.container-image.build=true
     ```
 
 2. Start the docker compose group by running the following command from the root of the repository:

getting-started/trino/README.md

@@ -24,7 +24,10 @@ This getting started guide provides a `docker-compose` file to set up [Trino](ht
 ## Build Polaris Image
 Build Polaris Image while Docker is running
 ```
-./gradlew clean :polaris-quarkus-server:assemble -Dquarkus.container-image.build=true --no-build-cache
+./gradlew \
+    :polaris-quarkus-server:assemble \
+    :polaris-quarkus-server:quarkusAppPartsBuild --rerun \
+    -Dquarkus.container-image.build=true
 ```
 
 ## Run the `docker-compose` file

gradle/libs.versions.toml

@@ -92,7 +92,7 @@ prometheus-metrics-exporter-servlet-jakarta = { module = "io.prometheus:promethe
 quarkus-bom = { module = "io.quarkus.platform:quarkus-bom", version.ref = "quarkus" }
 scala212-lang-library = { module = "org.scala-lang:scala-library", version.ref = "scala212" }
 scala212-lang-reflect = { module = "org.scala-lang:scala-reflect", version.ref = "scala212" }
-s3mock-testcontainers = { module = "com.adobe.testing:s3mock-testcontainers", version = "3.12.0" }
+s3mock-testcontainers = { module = "com.adobe.testing:s3mock-testcontainers", version = "4.0.0" }
 slf4j-api = { module = "org.slf4j:slf4j-api", version.ref = "slf4j" }
 smallrye-common-annotation = { module = "io.smallrye.common:smallrye-common-annotation", version = "2.11.0" }
 smallrye-config-core = { module = "io.smallrye.config:smallrye-config-core", version = "3.12.4" }