Dremio merge 2025 08 07 12 36 (apache#101)

* fix(deps): update dependency com.nimbusds:nimbus-jose-jwt to v10.4.1 (apache#2270)

* chore(deps): update actions/download-artifact action to v5 (apache#2271)

* fix(deps): update dependency boto3 to v1.40.3 (apache#2269)

* Prefer diagnostics field in Resolver (apache#2247)

* Stop mocking PolarisDiagnostics (apache#2248)

if diagnostis checks are failing in our tests we want to know about it

* Add TestServices.newCallContext (apache#2249)

also add local `newCallContext` helper in some test classes

* Nit: simplify runtime-service dependencies (apache#2273)

Dependency "io.quarkus:quarkus-jdbc-postgresql" doesn't need any excludes (these excludes were for `SparkIT` which is now isolated in a separate module).

* Minor fixes and enhancements to External IDP documentation (apache#2274)

* Standardize logging libraries in tests (apache#2268)

This change enforces the following test logging patterns:

- Non-Quarkus modules use Logback Classic, configured via logback-test.xml
- Quarkus modules use JBoss Logging Manager, configured in Quarkus configuration files.

This change also introduces a workaround for the "duplicate log messages" issues with Gradle + JBoss Logging Manager. See this issue for context:

quarkusio/quarkus#22844

The workaround implemented in this PR is very similar to the one proposed in this comment:

quarkusio/quarkus#22844 (comment)

Note: it's not entirely possible imho to suppress the following message on the console:

```
The Agroal dependency is present but no JDBC datasources have been defined.
```

This is because:

1. The message happens during augmentation phase, not during tests
2. And it suffers from the "duplicate message" issue with (it's actually Gradle that prints those messages).

* Use Mockito Java agent for mock instrumentation (apache#2275)

This change fixes the following warning during tests:

    Mockito is currently self-attaching to enable the inline-mock-maker. This will no longer work in future releases of the JDK. Please add Mockito as an agent to your build as described in Mockito's documentation: https://javadoc.io/doc/org.mockito/mockito-core/latest/org.mockito/org/mockito/Mockito.html#0.3
    WARNING: A Java agent has been loaded dynamically (.../byte-buddy-agent-1.17.5.jar)
    WARNING: If a serviceability tool is in use, please run with -XX:+EnableDynamicAgentLoading to hide this warning
    WARNING: If a serviceability tool is not in use, please run with -Djdk.instrument.traceUsage for more information
    WARNING: Dynamic loading of agents will be disallowed by default in a future release

* Use injected PolarisDiagnostics in MetaStoreManagerFactory impls (apache#2251)

* Clean exit when running repair mode for client (apache#2287)

* Clean exit when running repair mode for client

* Clean exit when running repair mode for client

* chore(deps): update dependency poetry to v2.1.4 (apache#2259)

* chore(deps): update dependency poetry to v2.1.4

* fix pyproject

---------

Co-authored-by: Robert Stupp <snazy@snazy.de>

* fix(deps): update dependency com.gradleup.shadow:shadow-gradle-plugin to v9 (apache#2289)

* chore(deps): update docker.io/jaegertracing/all-in-one docker tag to v1.72.0 (apache#2285)

* fix(deps): update dependency boto3 to v1.40.4 (apache#2284)

* Remove PolarisCallContext.getClock (apache#2250)

the clock is application scoped and thus should not be put into any
realm or call specific context class.

* Add PolarisAdminService.loadEntities helper (apache#2261)

`PolarisAdminService` has multiple spots where it is working around the
sub-optimal `PolarisMetaStoreManager` APIs.

This results in multiple fixes like apache#1949 and apache#2258

While eventually the underlying APIs should be improved, for now we can
make a single central workaround and clean up some redundant code.
Also we can improve the return types as callers are not interested in
details of the entity layer.

* fix(deps): update dependency com.google.cloud:google-cloud-storage-bom to v2.55.0 (apache#2281)

* fix: typo in server template files. (apache#2288)

* NoSQL: merge related adoptions

* Last merged commit d753e3d

---------

Co-authored-by: Mend Renovate <bot@renovateapp.com>
Co-authored-by: Christopher Lambert <xn137@gmx.de>
Co-authored-by: Alexandre Dutra <adutra@apache.org>
Co-authored-by: Yong Zheng <yongzheng0809@gmail.com>
Co-authored-by: Yujiang Zhong <42907416+zhongyujiang@users.noreply.github.com>

Author: 6 people

.github/actions/ci-incr-build-cache-prepare/action.yml

@@ -55,7 +55,7 @@ runs:
       run: ./gradlew -h
 
     - name: Download existing workflow artifacts
-      uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
+      uses: actions/download-artifact@634f93cb2916e3fdff6788551b99b062d0335ce0 # v5
       # Just in case, don't know the exact inner workings of Gradle's build cache and whether
       # the download-action complains about duplicate files.
       continue-on-error: true

build-logic/src/main/kotlin/polaris-java.gradle.kts

@@ -107,11 +107,13 @@ testing {
       dependencies {
         implementation(project())
         implementation(testFixtures(project()))
-        runtimeOnly(
-          libs.findLibrary("logback-classic").orElseThrow {
-            GradleException("logback-classic not declared in libs.versions.toml")
-          }
-        )
+        if (!plugins.hasPlugin("io.quarkus")) {
+          implementation(
+            libs.findLibrary("logback-classic").orElseThrow {
+              GradleException("logback-classic not declared in libs.versions.toml")
+            }
+          )
+        }
         implementation(
           libs.findLibrary("assertj-core").orElseThrow {
             GradleException("assertj-core not declared in libs.versions.toml")
@@ -138,6 +140,8 @@ testing {
   }
 }
 
+val mockitoAgent = configurations.create("mockitoAgent")
+
 dependencies {
   val libs = versionCatalogs.named("libs")
   testFixturesImplementation(
@@ -153,18 +157,24 @@ dependencies {
       GradleException("assertj-core not declared in libs.versions.toml")
     }
   )
-  testFixturesImplementation(
+  val mockitoCoreLib =
     libs.findLibrary("mockito-core").orElseThrow {
       GradleException("mockito-core not declared in libs.versions.toml")
     }
-  )
+
+  testFixturesImplementation(mockitoCoreLib)
+
+  mockitoAgent(mockitoCoreLib) { isTransitive = false }
 }
 
 tasks.withType<Test>().configureEach {
   systemProperty("file.encoding", "UTF-8")
   systemProperty("user.language", "en")
   systemProperty("user.country", "US")
   systemProperty("user.variant", "")
+  jvmArgumentProviders.add(
+    CommandLineArgumentProvider { listOf("-javaagent:${mockitoAgent.asPath}") }
+  )
 }
 
 tasks.withType<Jar>().configureEach {

build-logic/src/main/kotlin/polaris-runtime.gradle.kts

@@ -24,6 +24,14 @@ testing {
   suites {
     withType<JvmTestSuite> {
       targets.all {
+        testTask.configure {
+          systemProperty("java.util.logging.manager", "org.jboss.logmanager.LogManager")
+          // Enable automatic extension detection to execute GradleDuplicateLoggingWorkaround
+          // automatically.
+          // See https://github.com/quarkusio/quarkus/issues/22844
+          systemProperty("junit.jupiter.extensions.autodetection.enabled", "true")
+        }
+
         if (testTask.name != "test") {
           testTask.configure {
             // For Quarkus...
@@ -47,6 +55,23 @@ testing {
   }
 }
 
+dependencies {
+  // All Quarkus projects should use JBoss LogManager with SLF4J, instead of Logback
+  implementation("org.jboss.slf4j:slf4j-jboss-logmanager")
+}
+
+configurations.all {
+  // Validate that Logback dependencies are not used in Quarkus modules.
+  dependencies.configureEach {
+    if (group == "ch.qos.logback") {
+      throw GradleException(
+        "Logback dependencies are not allowed in Quarkus modules. " +
+          "Found $group:$name in ${project.name}."
+      )
+    }
+  }
+}
+
 // Let the test's implementation config extend testImplementation, so it also inherits the
 // project's "main" implementation dependencies (not just the "api" configuration)
 configurations.named("intTestImplementation").configure {

client/python/pyproject.toml

@@ -34,7 +34,7 @@ dependencies = [
     "python-dateutil>=2.8.2",
     "pydantic>=2.0.0",
     "typing-extensions>=4.7.1",
-    "boto3==1.40.2",
+    "boto3==1.40.4",
 ]
 
 [project.urls]
@@ -45,7 +45,7 @@ repository = "https://github.com/apache/polaris/"
 polaris = "cli.polaris_cli:main"
 
 [tool.poetry]
-requires-poetry = "==2.1.3"
+requires-poetry = "==2.1.4"
 packages = [
     { include = "polaris" },
     { include = "cli" }

getting-started/telemetry/docker-compose.yml

@@ -85,7 +85,7 @@ services:
 
   # Jaeger (OpenTelemetry traces collector)
   jaeger:
-    image: docker.io/jaegertracing/all-in-one:1.71.0
+    image: docker.io/jaegertracing/all-in-one:1.72.0
     ports:
       # Jaeger gRPC collector, used by Polaris
       - "4317:4317"

gradle/baselibs.versions.toml

@@ -22,5 +22,5 @@ errorprone = { module = "net.ltgt.gradle:gradle-errorprone-plugin", version = "4
 idea-ext = { module = "gradle.plugin.org.jetbrains.gradle.plugin.idea-ext:gradle-idea-ext", version = "1.2" }
 license-report = { module = "com.github.jk1:gradle-license-report", version = "2.9" }
 nexus-publish = { module = "io.github.gradle-nexus:publish-plugin", version = "2.0.0" }
-shadow = { module = "com.gradleup.shadow:shadow-gradle-plugin", version = "8.3.9" }
+shadow = { module = "com.gradleup.shadow:shadow-gradle-plugin", version = "9.0.0" }
 spotless = { module = "com.diffplug.spotless:spotless-plugin-gradle", version = "7.2.1" }

gradle/libs.versions.toml

@@ -53,7 +53,7 @@ commons-text = { module = "org.apache.commons:commons-text", version = "1.14.0"
 docker-java-api = { module = "com.github.docker-java:docker-java-api", version = "3.5.3" }
 eclipselink = { module = "org.eclipse.persistence:eclipselink", version = "4.0.7" }
 errorprone = { module = "com.google.errorprone:error_prone_core", version = "2.41.0" }
-google-cloud-storage-bom = { module = "com.google.cloud:google-cloud-storage-bom", version = "2.54.0" }
+google-cloud-storage-bom = { module = "com.google.cloud:google-cloud-storage-bom", version = "2.55.0" }
 guava = { module = "com.google.guava:guava", version = "33.4.8-jre" }
 h2 = { module = "com.h2database:h2", version = "2.3.232" }
 dnsjava = { module = "dnsjava:dnsjava", version = "3.6.3" }

persistence/eclipselink/src/main/java/org/apache/polaris/extension/persistence/impl/eclipselink/EclipseLinkPolarisMetaStoreManagerFactory.java

@@ -24,6 +24,7 @@
 import jakarta.enterprise.context.ApplicationScoped;
 import jakarta.inject.Inject;
 import java.nio.file.Path;
+import java.time.Clock;
 import org.apache.polaris.core.PolarisDiagnostics;
 import org.apache.polaris.core.context.RealmContext;
 import org.apache.polaris.core.persistence.LocalPolarisMetaStoreManagerFactory;
@@ -45,13 +46,14 @@ public class EclipseLinkPolarisMetaStoreManagerFactory
   @Inject EclipseLinkConfiguration eclipseLinkConfiguration;
   @Inject PolarisStorageIntegrationProvider storageIntegrationProvider;
 
+  @SuppressWarnings("unused") // Required by CDI
   protected EclipseLinkPolarisMetaStoreManagerFactory() {
-    this(null);
+    this(null, null);
   }
 
   @Inject
-  protected EclipseLinkPolarisMetaStoreManagerFactory(PolarisDiagnostics diagnostics) {
-    super(diagnostics);
+  protected EclipseLinkPolarisMetaStoreManagerFactory(Clock clock, PolarisDiagnostics diagnostics) {
+    super(clock, diagnostics);
   }
 
   @Override

persistence/eclipselink/src/test/java/org/apache/polaris/extension/persistence/impl/eclipselink/PolarisEclipseLinkMetaStoreManagerTest.java

@@ -30,13 +30,11 @@
 import java.net.URL;
 import java.nio.file.Files;
 import java.nio.file.Path;
-import java.time.ZoneId;
 import java.util.Objects;
 import java.util.stream.Stream;
 import org.apache.polaris.core.PolarisCallContext;
 import org.apache.polaris.core.PolarisDefaultDiagServiceImpl;
 import org.apache.polaris.core.PolarisDiagnostics;
-import org.apache.polaris.core.config.PolarisConfigurationStore;
 import org.apache.polaris.core.context.RealmContext;
 import org.apache.polaris.core.entity.PolarisPrincipalSecrets;
 import org.apache.polaris.core.persistence.BasePolarisMetaStoreManagerTest;
@@ -89,14 +87,10 @@ protected PolarisTestMetaStoreManager createPolarisTestMetaStoreManager() {
     PolarisEclipseLinkMetaStoreSessionImpl session =
         new PolarisEclipseLinkMetaStoreSessionImpl(
             store, Mockito.mock(), realmContext, null, "polaris", RANDOM_SECRETS);
-    return new PolarisTestMetaStoreManager(
-        new TransactionalMetaStoreManagerImpl(),
-        new PolarisCallContext(
-            realmContext,
-            session,
-            diagServices,
-            new PolarisConfigurationStore() {},
-            timeSource.withZone(ZoneId.systemDefault())));
+    TransactionalMetaStoreManagerImpl metaStoreManager =
+        new TransactionalMetaStoreManagerImpl(clock);
+    PolarisCallContext callCtx = new PolarisCallContext(realmContext, session, diagServices);
+    return new PolarisTestMetaStoreManager(metaStoreManager, callCtx);
   }
 
   @ParameterizedTest

persistence/nosql/persistence/metastore/src/main/java/org/apache/polaris/persistence/nosql/metastore/PersistenceMetaStoreManager.java

@@ -23,6 +23,7 @@
 
 import jakarta.annotation.Nonnull;
 import jakarta.annotation.Nullable;
+import java.time.Clock;
 import java.util.List;
 import java.util.Map;
 import java.util.Optional;
@@ -70,14 +71,17 @@ class PersistenceMetaStoreManager implements PolarisMetaStoreManager {
   private final Supplier<BaseResult> purgeRealm;
   private final RootCredentialsSet rootCredentialsSet;
   private final Supplier<PersistenceMetaStore> metaStoreSupplier;
+  private final Clock clock;
 
   PersistenceMetaStoreManager(
       Supplier<BaseResult> purgeRealm,
       RootCredentialsSet rootCredentialsSet,
-      Supplier<PersistenceMetaStore> metaStoreSupplier) {
+      Supplier<PersistenceMetaStore> metaStoreSupplier,
+      Clock clock) {
     this.purgeRealm = purgeRealm;
     this.rootCredentialsSet = rootCredentialsSet;
     this.metaStoreSupplier = metaStoreSupplier;
+    this.clock = clock;
   }
 
   PersistenceMetaStore ms() {
@@ -525,7 +529,7 @@ public EntitiesResult loadTasks(
                           PolarisTaskConstants.TASK_TIMEOUT_MILLIS);
               return taskState == null
                   || taskState.executor == null
-                  || callCtx.getClock().millis() - taskState.lastAttemptStartTime > taskAgeTimeout;
+                  || clock.millis() - taskState.lastAttemptStartTime > taskAgeTimeout;
             },
             Function.identity(),
             PageToken.readEverything());
@@ -540,8 +544,7 @@ public EntitiesResult loadTasks(
               var properties = PolarisObjectMapperUtil.deserializeProperties(task.getProperties());
               properties.put(PolarisTaskConstants.LAST_ATTEMPT_EXECUTOR_ID, executorId);
               properties.put(
-                  PolarisTaskConstants.LAST_ATTEMPT_START_TIME,
-                  String.valueOf(callCtx.getClock().millis()));
+                  PolarisTaskConstants.LAST_ATTEMPT_START_TIME, String.valueOf(clock.millis()));
               properties.put(
                   PolarisTaskConstants.ATTEMPT_COUNT,
                   String.valueOf(