Skip to content

Commit f866828

Browse files
snazypoojanilangekareric-maynardrenovate-botMonkeyCanCode
authored
Dremio merge 2025 09 08 13 38 (apache#110)
* Integration tests for Catalog Federation (apache#2344) Adds a Junit5 integration test for catalog federation. * Fix merge conflict in CatalogFederationIntegrationTest (apache#2420) apache#2344 added a new test for catalog federation, but it looks like an undetected conflict with concurrent changes related to authentication have broken the test in main. * chore(deps): update registry.access.redhat.com/ubi9/openjdk-21-runtime docker tag to v1.23-6.1755674729 (apache#2416) * 2334 (apache#2427) * Fix TableIdentifier in TaskFileIOSupplier (apache#2304) we cant just convert a `TaskEntity` to a `IcebergTableLikeEntity` as the `getTableIdentifier()` method will not return a correct value by using the name of the task and its parent namespace (which is empty?). task handlers instead need to pass in the `TableIdentifier` that they already inferred via `TaskEntity.readData`. * Fix NPE in CreateCatalog (apache#2435) * Doc fix: Access control page update (apache#2424) * 2418 * 2418 * fix(deps): update dependency software.amazon.awssdk:bom to v2.32.29 (apache#2443) * Optimize PolicyCatalog.listPolicies (apache#2370) this is a follow-up to apache#2290 the optimization is to use `listEntities` instead of `loadEntities` when there is `policyType` filter to apply * Add PolarisDiagnostics field to BaseMetaStoreManager (apache#2381) * Add PolarisDiagnostics field to BaseMetaStoreManager the ultimate goal is removing the `PolarisCallContext` parameter from every `PolarisMetaStoreManager` interface method, so we make steps towards reducing its usage first. * Add feature flag to disallow custom S3 endpoints (apache#2442) * Add new realm-level flag: `ALLOW_SETTING_S3_ENDPOINTS` (default: true) * Enforce in `PolarisServiceImpl.validateStorageConfig()` Fixes apache#2436 * Deprecate ActiveRolesProvider for removal (apache#2404) * Client: fix openapi verbose output, remove doc generate, and skip test generations (apache#2439) * Fix various issue in client code generation * Use logger instead of print * Add back exclude on __pycache__ as CI is not via Makefile * Add back exclude on __pycache__ as CI is not via Makefile * Add user principal tag in metrics (apache#2445) * Added API change to enable tag * Added test * Added production readiness check * fix(deps): update dependency io.opentelemetry.semconv:opentelemetry-semconv to v1.36.0 (apache#2454) * fix(deps): update dependency com.google.cloud:google-cloud-storage-bom to v2.56.0 (apache#2447) * fix(deps): update dependency gradle.plugin.org.jetbrains.gradle.plugin.idea-ext:gradle-idea-ext to v1.3 (apache#2428) * Build: Make jandex dependency used for index generation managed (apache#2431) Also allows specifying the jandex index version for the build. This is a preparation step contributing to apache#2204, once a jandex fix for reproducible builds is available. Co-authored-by: Alexandre Dutra <adutra@apache.org> * Built: improve reproducible archive files (apache#2432) As part of the effort for apache#2204, this change fixes a few aspects around reproducible builds: Some Gradle projects produce archive files, but don't get the necessary Gradle archive-tasks settings applied: one not-published project but also the tarball&zip of the distribution. This change moves the logic to the new build-plugin `polaris-reproducible`. Another change is to have some Quarkus generated jar files adhere to the same conventions, which are constant timestamps for the zip entries and a deterministic order of the entries. That's sadly not a full fix, as the classes that are generated or instumented by Quarkus differ in each build. Contributes to apache#2204 * Remove commons-lang3 dependency (apache#2456) outside of tests we can replace the functionality with jdk11 and guava. also stop using `org.assertj.core.util` as its a non-public api. * add refresh credentials property to loadTableResult (apache#2341) * add refresh credentials property to loadTableResult * IcebergCatalogAdapterTest: Added test to ensure refresh credentials endpoint is included * delegate refresh credential endpoint configuration to storage integration * GCP: Add refresh credential properties * fix(deps): update dependency io.opentelemetry.semconv:opentelemetry-semconv to v1.37.0 (apache#2458) * Add Delegator to all API Implementations (apache#2434) Per the Dev ML, implements the Delegator pattern to add Events instrumentation to all Polaris APIs. * Prefer java.util.Base64 over commons-codec (apache#2463) `java.util.Base64` is available since java8 and we are already using it in a few other spots. in a follow-up we might be able to get rid of our `commons-codec` dependency completely. * Service: Move tests to the right package (apache#2469) * Update versions in runtime LICENSE and NOTICE (apache#2468) * fix(deps): update dependency com.adobe.testing:s3mock-testcontainers to v4.8.0 (apache#2475) * fix(deps): update dependency com.gradleup.shadow:shadow-gradle-plugin to v9.1.0 (apache#2476) * Service: Remove hadoop-common from polaris-runtime-service (apache#2462) * Service: Always validate allowed locations from Storage Config (apache#2473) * Add Community Sync Meeting 20250828 (apache#2477) * Update dependency software.amazon.awssdk:bom to v2.33.0 (apache#2483) * Remove PolarisCallContext.getDiagServices (apache#2415) * Remove PolarisCallContext.getDiagServices usage * Remove diagnostics from PolarisCallContext * Feature: Expose resetCredentials via a new reset api to allow root user to reset credentials for an existing principal with custom values (apache#2197) * Add type-check to PolarisEntity subclass ctors (apache#2302) currently one can freely "cast" any `PolarisEntity` to a more specific type via their constructors. this can lead to subtle bugs like we fixed in a29f800 by adding type checks we discover a few more places where we need to be more careful about how we construct new or handle existing entities. note that we can add a check for `PolarisEntitySubType` in a followup, but it requires more fixes currently. * Fix CI (apache#2489) Fix undetected merge conflict after apache#2197 + apache#2415 + apache#2434 * Use local diagnostics in TransactionWorkspaceMetaStoreManager * Add resetCredentials to PolarisPrincipalsEventServiceDelegator * Core: Prevent AIOOBE for negative codes in PolarisEntityType, PolarisPrivilege, ReturnStatus (apache#2490) * feat(idgen): Start Implementation of NoSQL with the ID Generation Framework (apache#2131) Create an ID Generation Framework. Related to apache#650 & apache#844 Co-authored-by: Robert Stupp <snazy@snazy.de> Co-authored-by: Dmitri Bourlatchkov <dmitri.bourlatchkov@gmail.com> * perf(refactor): optimizing JdbcBasePersistenceImpl.listEntities (apache#2465) - Reduced Column Selection: Only 6 columns instead of 16 - Eliminated Object Creation Overhead: Direct conversion to EntityNameLookupRecord without intermediate PolarisBaseEntity * Add Polaris Events to Persistence (apache#1844) * AWS CloudWatch Event Sink Implementation (apache#1965) * Fix failing CI (apache#2498) * Update actions/stale digest to 3a9db7e (apache#2499) * Core: Prevent AIOOBE for negative policy codes in PredefinedPolicyType (apache#2486) * Service: Add location tests for views (apache#2496) * Update docker.io/jaegertracing/all-in-one Docker tag to v1.73.0 (apache#2500) * Update dependency io.netty:netty-codec-http2 to v4.2.5.Final (apache#2495) * Update actions/setup-python action to v6 (apache#2502) * Update the Release Guide about the Helm Chart package (apache#2179) * Update the Release Guide about the Helm Chart package * Update release-guide.md Co-authored-by: Pierre Laporte <pierre@pingtimeout.fr> * Add missing commit message * Whitespace * Use Helm GPG plugin to sign the Helm chart * Fix directories during Helm chart copy to SVN * Add Helm index to SVN * Use long name for svn checkout * Ensure the Helm index is updated after the chart is moved to SVN dist release * Do not publish any Docker image before the vote succeeds * Typos * Revert "Do not publish any Docker image before the vote succeeds" This reverts commit 5617e65. * Don't mention Helm values.yaml in the release guide as it doesn't contain version details --------- Co-authored-by: Pierre Laporte <pierre@pingtimeout.fr> * Update dependency com.azure:azure-sdk-bom to v1.2.38 (apache#2503) * Update registry.access.redhat.com/ubi9/openjdk-21-runtime Docker tag to v1.23-6.1756793420 (apache#2504) * Remove commons-codec dependency (apache#2474) follow-up to f8ad77a we can simply use guava instead and eliminate the extra dependency * CLI: Remove SCRIPT_DIR and default config location to user home (apache#2448) * Remove readInternalProperties helpers (apache#2506) the functionality is already provided by the `PrincipalEntity` * Add Events for Generic Table APIs (apache#2481) This PR adds the Events instrumentation for the Generic Tables Service APIs, surrounding the default delegated call to the business logic APIs. * Disable custom namespace locations (apache#2422) When we create a namespace or alter its location, we must confirm that this location is within the parent location. This PR introduces introduces a check similar to the one we have for tables, where custom locations are prohibited by default. This functionality is gated behind a new behavior change flag `ALLOW_NAMESPACE_CUSTOM_LOCATION`. In addition to allowing us to revert to the old behavior, this flag allows some tests relying on arbitrarily-located namespaces to pass (such as those from upstream Iceberg). Fixes: apache#2417 * fix for IcebergAllowedLocationTest (apache#2511) * Remove unused config from SparkSessionBuilder (apache#2512) Tests pass without it. * Add Events for Policy Service APIs (apache#2479) * Remove PolarisTestMetaStoreManager.jsonNode helper (apache#2513) * Update dependency software.amazon.awssdk:bom to v2.33.4 (apache#2517) * Update dependency com.nimbusds:nimbus-jose-jwt to v10.5 (apache#2514) * Update dependency io.opentelemetry:opentelemetry-bom to v1.54.0 (apache#2515) * Update dependency io.micrometer:micrometer-bom to v1.15.4 (apache#2519) * Port missed OSS change * NoSQL: adopt to updated test packages * NoSQL: adapt to removed PolarisDiagnostics param * NoSQL: fix libs.versions.toml * NoSQL: include jandex plugin related changes from OSS * NoSQL: changes for delete/set principal client-ID+secret * Last merged commit c6176dc --------- Co-authored-by: Pooja Nilangekar <poojan@umd.edu> Co-authored-by: Eric Maynard <eric.maynard+oss@snowflake.com> Co-authored-by: Mend Renovate <bot@renovateapp.com> Co-authored-by: Yong Zheng <yongzheng0809@gmail.com> Co-authored-by: Christopher Lambert <xn137@gmx.de> Co-authored-by: Honah (Jonas) J. <honahx@apache.org> Co-authored-by: Dmitri Bourlatchkov <dmitri.bourlatchkov@gmail.com> Co-authored-by: Alexandre Dutra <adutra@apache.org> Co-authored-by: fivetran-kostaszoumpatianos <kostas.zoumpatianos@fivetran.com> Co-authored-by: Jason <jasonf20@gmail.com> Co-authored-by: Adnan Hemani <adnan.h@berkeley.edu> Co-authored-by: Yufei Gu <yufei@apache.org> Co-authored-by: JB Onofré <jbonofre@apache.org> Co-authored-by: fivetran-arunsuri <103934371+fivetran-arunsuri@users.noreply.github.com> Co-authored-by: Adam Christian <105929021+adam-christian-software@users.noreply.github.com> Co-authored-by: Artur Rakhmatulin <artur.rakhmatulin@gmail.com> Co-authored-by: Pierre Laporte <pierre@pingtimeout.fr>
1 parent ba9fa4b commit f866828

File tree

327 files changed

+8280
-1773
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

327 files changed

+8280
-1773
lines changed

.github/workflows/python-client.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,7 @@ jobs:
5151
java-version: '21'
5252

5353
- name: Set up Python ${{ matrix.python-version }}
54-
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
54+
uses: actions/setup-python@e797f83bcb11b83ae66e0230d6156d7c80228e7c # v6
5555
with:
5656
python-version: ${{ matrix.python-version }}
5757

.github/workflows/stale.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ jobs:
2222
stale:
2323
runs-on: ubuntu-24.04
2424
steps:
25-
- uses: actions/stale@8f717f0dfca33b78d3c933452e42558e4456c8e7
25+
- uses: actions/stale@3a9db7e6a41a89f618792c92c0e97cc736e1b13f
2626
with:
2727
days-before-close: 5
2828
days-before-stale: 30

CHANGELOG.md

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,8 @@ request adding CHANGELOG notes for breaking (!) changes and possibly other secti
5151
### New Features
5252

5353
- Added Catalog configuration for S3 and STS endpoints. This also allows using non-AWS S3 implementations.
54+
The realm-level feature flag `ALLOW_SETTING_S3_ENDPOINTS` (default: true) may be used to disable this
55+
functionality.
5456

5557
- The `IMPLICIT` authentication type enables users to create federated catalogs without explicitly
5658
providing authentication parameters to Polaris. When the authentication type is set to `IMPLICIT`,
@@ -81,6 +83,13 @@ at locations that better optimize for object storage.
8183

8284
- Introduced bootstrap command options to specify custom schema files for database initialization.
8385

86+
- Added refresh credentials endpoint configuration to LoadTableResponse for AWS, Azure, and GCP. Enabling
87+
automatic storage credential refresh per table on the client side. Java client version >= 1.8.0 is required.
88+
The endpoint path is always returned when using vended credentials, but clients must enable the
89+
refresh-credentials flag for the desired storage provider.
90+
91+
- Added a Management API endpoint to reset principal credentials, controlled by the `ENABLE_CREDENTIAL_RESET` (default: true) feature flag.
92+
8493
### Changes
8594

8695
- Polaris Management API clients must be prepared to deal with new attributes in `AwsStorageConfigInfo` objects.
@@ -89,9 +98,8 @@ at locations that better optimize for object storage.
8998

9099
### Deprecations
91100

92-
* The property `polaris.active-roles-provider.type` is deprecated in favor of
93-
`polaris.authentication.active-roles-provider.type`. The old property is still supported, but will be removed in a
94-
future release.
101+
- The property `polaris.active-roles-provider.type` is deprecated for removal.
102+
- The `ActiveRolesProvider` interface is deprecated for removal.
95103

96104
### Fixes
97105

Makefile

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -134,13 +134,13 @@ client-lint: client-setup-env ## Run linting checks for Polaris client
134134
.PHONY: client-regenerate
135135
client-regenerate: client-setup-env ## Regenerate the client code
136136
@echo "--- Regenerating client code ---"
137-
@$(ACTIVATE_AND_CD) && python3 generate_clients.py
137+
@$(ACTIVATE_AND_CD) && python3 -B generate_clients.py
138138
@echo "--- Client code regeneration complete ---"
139139

140140
.PHONY: client-unit-test
141141
client-unit-test: client-setup-env ## Run client unit tests
142142
@echo "--- Running client unit tests ---"
143-
@$(ACTIVATE_AND_CD) && SCRIPT_DIR="non-existing-mock-directory" poetry run pytest test/
143+
@$(ACTIVATE_AND_CD) && poetry run pytest test/
144144
@echo "--- Client unit tests complete ---"
145145

146146
.PHONY: client-integration-test

api/iceberg-service/build.gradle.kts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ import org.openapitools.generator.gradle.plugin.tasks.GenerateTask
2222
plugins {
2323
alias(libs.plugins.openapi.generator)
2424
id("polaris-client")
25-
alias(libs.plugins.jandex)
25+
id("org.kordamp.gradle.jandex")
2626
}
2727

2828
dependencies {

api/management-model/build.gradle.kts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ import org.openapitools.generator.gradle.plugin.tasks.GenerateTask
2222
plugins {
2323
alias(libs.plugins.openapi.generator)
2424
id("polaris-client")
25-
alias(libs.plugins.jandex)
25+
id("org.kordamp.gradle.jandex")
2626
}
2727

2828
dependencies {

api/management-service/build.gradle.kts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ import org.openapitools.generator.gradle.plugin.tasks.GenerateTask
2222
plugins {
2323
alias(libs.plugins.openapi.generator)
2424
id("polaris-client")
25-
alias(libs.plugins.jandex)
25+
id("org.kordamp.gradle.jandex")
2626
}
2727

2828
dependencies {

api/polaris-catalog-service/build.gradle.kts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ import org.openapitools.generator.gradle.plugin.tasks.GenerateTask
2222
plugins {
2323
alias(libs.plugins.openapi.generator)
2424
id("polaris-client")
25-
alias(libs.plugins.jandex)
25+
id("org.kordamp.gradle.jandex")
2626
}
2727

2828
val genericTableModels =

build-logic/build.gradle.kts

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ dependencies {
2323
implementation(gradleKotlinDsl())
2424
implementation(baselibs.errorprone)
2525
implementation(baselibs.idea.ext)
26+
implementation(baselibs.jandex)
2627
implementation(baselibs.license.report)
2728
implementation(baselibs.nexus.publish)
2829
implementation(baselibs.shadow)

build-logic/src/main/kotlin/Utilities.kt

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,11 @@
1414
* limitations under the License.
1515
*/
1616

17+
import java.io.File
18+
import java.io.FileOutputStream
19+
import java.nio.file.attribute.FileTime
20+
import java.util.zip.ZipFile
21+
import java.util.zip.ZipOutputStream
1722
import org.gradle.api.Project
1823
import org.gradle.process.JavaForkOptions
1924

@@ -61,3 +66,41 @@ fun JavaForkOptions.addSparkJvmOptions() {
6166
"-Djdk.reflect.useDirectMethodHandle=false",
6267
)
6368
}
69+
70+
/**
71+
* Rewrites the given ZIP file.
72+
*
73+
* The timestamps of all entries are set to `1980-02-01 00:00`, zip entries appear in a
74+
* deterministic order.
75+
*/
76+
fun makeZipReproducible(source: File) {
77+
val t = FileTime.fromMillis(318211200_000) // 1980-02-01 00:00 GMT
78+
79+
val outFile = File(source.absolutePath + ".tmp.out")
80+
81+
val names = mutableListOf<String>()
82+
ZipFile(source).use { zip -> zip.stream().forEach { e -> names.add(e.name) } }
83+
names.sort()
84+
85+
ZipOutputStream(FileOutputStream(outFile)).use { dst ->
86+
ZipFile(source).use { zip ->
87+
names.forEach { n ->
88+
val e = zip.getEntry(n)
89+
zip.getInputStream(e).use { src ->
90+
e.setCreationTime(t)
91+
e.setLastAccessTime(t)
92+
e.setLastModifiedTime(t)
93+
dst.putNextEntry(e)
94+
src.copyTo(dst)
95+
dst.closeEntry()
96+
src.close()
97+
}
98+
}
99+
}
100+
}
101+
102+
val origFile = File(source.absolutePath + ".tmp.orig")
103+
source.renameTo(origFile)
104+
outFile.renameTo(source)
105+
origFile.delete()
106+
}

0 commit comments

Comments
 (0)