Skip to content

Commit f563dbc

Browse files
snazyHonahXgh-yzoumansehajsinghrenovate-bot
authored
Merge OSS main branch + adoptions (#51)
* Policy Store: PolicyMappingRecord with Persistence Impl (apache#1104) * Spark: Setup repository code structure and build (apache#1190) * Added freshness aware table loading using metadata file location for ETag (apache#1037) * Pulled in iceberg 1.8.0 spec changes for freshness aware table loading and added feature to Polaris * Changed etag support to use entityId:version tuple * fixed getresponse call * Changed etagged response to record and gave default implementation to ETaggableEntity * Made iceberg rest spec docs clearer * Added HTTP Compliant ETag and IfNoneMatch representations and separated persistence from etag logic * Changed ETag to be a record and improved semantics of IfNoneMatch * Fixed semantics of if none match * Removed ETag representation, consolidated in IfNoneMatch * fixed if none match parsing * Added table entity retrieval method to table operations * removed accidental commit of pycache folders * Fixed formatting * Changed to use metadata location hash * Ran formatting * use sha256 * Moved out ETag functions to utility class and removed ETaggedLoadTableResponse * Addressed comments * Fixed IcebergTableLikeEntity package rename * main: Update dependency io.opentelemetry.semconv:opentelemetry-semconv to v1.31.0 (apache#1288) * Update LICENSE and NOTICE in the distributions (admin and server) (apache#1258) * Gradle/Quarkus: make imageBuild task depend on jandex (apache#1290) * Core: Clarify the atomicity of BasePersistence methods (apache#1274) * Implement GenericTableCatalogAdapter (apache#1264) * rebase * more fixes * autolint * working on tests * stable test * autolint * polish * changes per review * some changes per review * grants * autolint * changes per review * changes per review * typofix * Improve code-containment and efficiency of etag-aware loading (apache#1296) * Improve code-containment and efficiency of etag-aware loading -Make the hash generation resilient against null metadataLocation -Use getResolvedPath instead of getPassthroughResolvedPath to avoid redundant persistence round-trip -Only try to calculate the etag for comparison against ifNoneMatch if the ifNoneMatch is actually provided * Add strict null-checking at callsites to generateETag, disallow passing null to generator * Add TODO to refactor shared logic for etag generation * Core: Add Endpoints and resource paths for Generic Table (apache#1286) * main: Update dependency com.nimbusds:nimbus-jose-jwt to v10.1 (apache#1299) * [JDBC] Part1 : ADD SQL script for Polaris setup (apache#1276) * main: Update registry.access.redhat.com/ubi9/openjdk-21-runtime Docker tag to v1.22-1.1743605859 (apache#1300) * done (apache#1297) * Add Polaris Community Meeting for April 3, 2025 (apache#1304) * Use config-file to define errorprone rule (apache#1233) Also enabled a couple more simple rules, and adding suppressions/fixes for/to the code. The two rules `EqualsGetClass` and `UnusedMethod`, which I think are useful, are not enabled yet, because that would mean actual code changes, which I do not want to do in this PR. The rule `PatternMatchingInstanceof`, introduced in apache#393, is disabled in this PR. It does not work before errorrpone 2.37.0 (via apache#1213) - requires additional changes to enable the rule (see apache#1215). * Add Yun as a contributor (apache#1310) * Refactor CatalogHandler to comply with ErrorProne rules (apache#1312) Fix the CI error after apache#1233 * Implement PolicyCatalog Stage 1: CRUD + ListPolicies (apache#1294) * main: Update dependency io.opentelemetry:opentelemetry-bom to v1.49.0 (apache#1316) * main: Update docker.io/jaegertracing/all-in-one Docker tag to v1.68.0 (apache#1317) * main: Update dependency boto3 to v1.37.28 (apache#1328) * main: Update dependency software.amazon.awssdk:bom to v2.31.16 (apache#1329) * Make `BasePolaritsMetaStoreManagerTest` and `(Base)ResolverTest` reusable (apache#1308) Moves the test cases into the `Base*` classes and make sure the classes can be reused by other persistence implementations. * main: Update dependency io.opentelemetry.semconv:opentelemetry-semconv to v1.32.0 (apache#1293) * main: Update mockito monorepo to v5.17.0 (apache#1311) * PySpark Update AWS Region (apache#1302) Co-authored-by: Travis Bowen <travis.bowen@snowflake.com> * main: Update dependency com.nimbusds:nimbus-jose-jwt to v10.2 (apache#1334) * main: Update dependency com.diffplug.spotless:spotless-plugin-gradle to v7.0.3 (apache#1335) * Maven publication: Produce correct `<scm><tag>` in `pom.xml` (apache#1330) `project.scm.tag` in a Maven pom is intended to refer to the SCM (Git) tag. We currently publish `main`, which is incorrect. This change omits the SCM tag for snapshot builds, but emits the Git tag for releases. * Remove `@StaticInitSafe` annotation (apache#1331) There was an issue around mapped configurations having the `@StaticInitSafe` annotation that led to _two_ instances (a "static" one and a "somewhet application-scoped" one) - this was fixed in Quarkus 3.21. One bug in smallrye-config is fixed for Quarkus > 3.21.0, another issue however remains. Since `@StaticInitSafe` annotated configs seem to cause some weird issues, it seems legit to remote that annotation altogether. This approach was [taken in Nessie](projectnessie/nessie#10606) as well. Investigations (via practical experiments) have proven that there's no measurable impact (runtime + heap) when doing this - and that's also been confirmed by Quarkus + Smallrye-config maintainers. Hence this change remotes that annotation from the code base. * Build/Release: Add a "generate digest" task and use for source tarball and Quarkus distributables (apache#1271) * Ensure that digest and signature are generated for both Polaris-Server and admin tar/zip distribution * Move "generate digest" functionality to a Gradle task * main: Update dependency com.google.errorprone:error_prone_core to v2.37.0 (apache#1213) * main: Update Quarkus Platform and Group to v3.21.1 (apache#1291) * main: Update dependency io.netty:netty-codec-http2 to v4.2.0.Final (apache#1301) * Adoptions after merge --------- Co-authored-by: Honah (Jonas) J. <honahx@apache.org> Co-authored-by: gh-yzou <167037035+gh-yzou@users.noreply.github.com> Co-authored-by: Mansehaj Singh <msehajs@gmail.com> Co-authored-by: Mend Renovate <bot@renovateapp.com> Co-authored-by: JB Onofré <jbonofre@apache.org> Co-authored-by: Alexandre Dutra <adutra@users.noreply.github.com> Co-authored-by: Yufei Gu <yufei@apache.org> Co-authored-by: Eric Maynard <eric.maynard+oss@snowflake.com> Co-authored-by: Dennis Huo <7410123+dennishuo@users.noreply.github.com> Co-authored-by: Prashant Singh <35593236+singhpk234@users.noreply.github.com> Co-authored-by: Travis Bowen <122238243+travis-bowen@users.noreply.github.com> Co-authored-by: Travis Bowen <travis.bowen@snowflake.com>
1 parent d116c7e commit f563dbc

File tree

58 files changed

+1544
-153
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

58 files changed

+1544
-153
lines changed

.asf.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,7 @@ github:
7272
# Adding renovate-bot as a collaborator, so CI doesn't need to be manually approved
7373
- renovate-bot
7474
- singhpk234
75+
- gh-yzou
7576

7677
notifications:
7778
commits: commits@polaris.apache.org

api/polaris-catalog-service/build.gradle.kts

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,6 @@ val policyManagementModels =
3636
"CatalogIdentifier",
3737
"CreatePolicyRequest",
3838
"LoadPolicyResponse",
39-
"PolicyIdentifier",
4039
"Policy",
4140
"PolicyAttachmentTarget",
4241
"AttachPolicyRequest",
@@ -59,6 +58,7 @@ dependencies {
5958
implementation(libs.jakarta.inject.api)
6059
implementation(libs.jakarta.validation.api)
6160
implementation(libs.swagger.annotations)
61+
implementation(libs.guava)
6262

6363
implementation(libs.jakarta.servlet.api)
6464
implementation(libs.jakarta.ws.rs.api)
@@ -103,6 +103,9 @@ openApiGenerate {
103103
"ErrorModel" to "org.apache.iceberg.rest.responses.ErrorResponse",
104104
"IcebergErrorResponse" to "org.apache.iceberg.rest.responses.ErrorResponse",
105105
"TableIdentifier" to "org.apache.iceberg.catalog.TableIdentifier",
106+
107+
// Custom types defined below
108+
"PolicyIdentifier" to "org.apache.polaris.service.types.PolicyIdentifier",
106109
)
107110
}
108111

Lines changed: 140 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,140 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one
3+
* or more contributor license agreements. See the NOTICE file
4+
* distributed with this work for additional information
5+
* regarding copyright ownership. The ASF licenses this file
6+
* to you under the Apache License, Version 2.0 (the
7+
* "License"); you may not use this file except in compliance
8+
* with the License. You may obtain a copy of the License at
9+
*
10+
* http://www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing,
13+
* software distributed under the License is distributed on an
14+
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
* KIND, either express or implied. See the License for the
16+
* specific language governing permissions and limitations
17+
* under the License.
18+
*/
19+
package org.apache.polaris.service.types;
20+
21+
import com.fasterxml.jackson.annotation.JsonCreator;
22+
import com.fasterxml.jackson.annotation.JsonProperty;
23+
import io.swagger.annotations.ApiModelProperty;
24+
import java.util.Objects;
25+
import org.apache.iceberg.catalog.Namespace;
26+
27+
/**
28+
* Represents a modified version of the PolicyIdentifier that is different from the one generated by
29+
* the OpenAPI generator
30+
*
31+
* <p>the open api generation inlines the namespace definition, generates a {@code List<String>}
32+
* directly, instead of generating a Namespace class. This version uses {@link
33+
* org.apache.iceberg.catalog.Namespace} for namespace field.
34+
*
35+
* <p>TODO: make code generation use {@link org.apache.iceberg.catalog.Namespace} directly
36+
*/
37+
public class PolicyIdentifier {
38+
39+
private final Namespace namespace;
40+
private final String name;
41+
42+
/** Reference to one or more levels of a namespace */
43+
@ApiModelProperty(
44+
example = "[\"accounting\",\"tax\"]",
45+
required = true,
46+
value = "Reference to one or more levels of a namespace")
47+
@JsonProperty(value = "namespace", required = true)
48+
public Namespace getNamespace() {
49+
return namespace;
50+
}
51+
52+
/** */
53+
@ApiModelProperty(required = true, value = "")
54+
@JsonProperty(value = "name", required = true)
55+
public String getName() {
56+
return name;
57+
}
58+
59+
@JsonCreator
60+
public PolicyIdentifier(
61+
@JsonProperty(value = "namespace", required = true) Namespace namespace,
62+
@JsonProperty(value = "name", required = true) String name) {
63+
this.namespace = Objects.requireNonNullElse(namespace, Namespace.empty());
64+
this.name = name;
65+
}
66+
67+
public static Builder builder() {
68+
return new Builder();
69+
}
70+
71+
public static Builder builder(Namespace namespace, String name) {
72+
return new Builder(namespace, name);
73+
}
74+
75+
public static final class Builder {
76+
private Namespace namespace;
77+
private String name;
78+
79+
private Builder() {}
80+
81+
private Builder(Namespace namespace, String name) {
82+
this.namespace = Objects.requireNonNullElse(namespace, Namespace.empty());
83+
this.name = name;
84+
}
85+
86+
public Builder setNamespace(Namespace namespace) {
87+
this.namespace = namespace;
88+
return this;
89+
}
90+
91+
public Builder setName(String name) {
92+
this.name = name;
93+
return this;
94+
}
95+
96+
public PolicyIdentifier build() {
97+
PolicyIdentifier inst = new PolicyIdentifier(namespace, name);
98+
return inst;
99+
}
100+
}
101+
102+
@Override
103+
public boolean equals(Object o) {
104+
if (this == o) {
105+
return true;
106+
}
107+
if (o == null || getClass() != o.getClass()) {
108+
return false;
109+
}
110+
PolicyIdentifier policyIdentifier = (PolicyIdentifier) o;
111+
return Objects.equals(this.namespace, policyIdentifier.namespace)
112+
&& Objects.equals(this.name, policyIdentifier.name);
113+
}
114+
115+
@Override
116+
public int hashCode() {
117+
return Objects.hash(namespace, name);
118+
}
119+
120+
@Override
121+
public String toString() {
122+
StringBuilder sb = new StringBuilder();
123+
sb.append("class PolicyIdentifier {\n");
124+
125+
sb.append(" namespace: ").append(toIndentedString(namespace)).append("\n");
126+
sb.append(" name: ").append(toIndentedString(name)).append("\n");
127+
sb.append("}");
128+
return sb.toString();
129+
}
130+
131+
/**
132+
* Convert the given object to string with each line indented by 4 spaces (except the first line).
133+
*/
134+
private String toIndentedString(Object o) {
135+
if (o == null) {
136+
return "null";
137+
}
138+
return o.toString().replace("\n", "\n ");
139+
}
140+
}

build-logic/src/main/kotlin/polaris-java.gradle.kts

Lines changed: 17 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,8 @@
1717
* under the License.
1818
*/
1919

20+
import java.util.Properties
21+
import net.ltgt.gradle.errorprone.CheckSeverity
2022
import net.ltgt.gradle.errorprone.errorprone
2123
import org.gradle.api.tasks.compile.JavaCompile
2224
import org.gradle.api.tasks.testing.Test
@@ -41,18 +43,23 @@ tasks.withType(JavaCompile::class.java).configureEach {
4143
options.errorprone.disableWarningsInGeneratedCode = true
4244
options.errorprone.excludedPaths =
4345
".*/${project.layout.buildDirectory.get().asFile.relativeTo(projectDir)}/generated/.*"
44-
options.errorprone.error(
45-
"DefaultCharset",
46-
"FallThrough",
47-
"MissingCasesInEnumSwitch",
48-
"MissingOverride",
49-
"ModifiedButNotUsed",
50-
"OrphanedFormatString",
51-
"PatternMatchingInstanceof",
52-
"StringCaseLocaleUsage",
53-
)
46+
val errorproneRules = rootProject.projectDir.resolve("codestyle/errorprone-rules.properties")
47+
inputs.file(errorproneRules).withPathSensitivity(PathSensitivity.RELATIVE)
48+
options.errorprone.checks.putAll(provider { memoizedErrorproneRules(errorproneRules) })
5449
}
5550

51+
private fun memoizedErrorproneRules(rulesFile: File): Map<String, CheckSeverity> =
52+
rulesFile.reader().use {
53+
val rules = Properties()
54+
rules.load(it)
55+
rules
56+
.mapKeys { e -> (e.key as String).trim() }
57+
.mapValues { e -> (e.value as String).trim() }
58+
.filter { e -> e.key.isNotEmpty() && e.value.isNotEmpty() }
59+
.mapValues { e -> CheckSeverity.valueOf(e.value) }
60+
.toMap()
61+
}
62+
5663
tasks.register("compileAll").configure {
5764
group = "build"
5865
description = "Runs all compilation and jar tasks"

build-logic/src/main/kotlin/publishing/PublishingHelperExtension.kt

Lines changed: 0 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -49,12 +49,6 @@ constructor(objectFactory: ObjectFactory, project: Project) {
4949
objectFactory
5050
.fileProperty()
5151
.convention(project.provider { distributionDir.get().file("${baseName.get()}.tar.gz") })
52-
val sourceTarballDigest =
53-
objectFactory
54-
.fileProperty()
55-
.convention(
56-
project.provider { distributionDir.get().file("${baseName.get()}.tar.gz.sha512") }
57-
)
5852

5953
val mailingLists = objectFactory.listProperty(String::class.java).convention(emptyList())
6054

build-logic/src/main/kotlin/publishing/configurePom.kt

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -103,7 +103,10 @@ internal fun configurePom(project: Project, mavenPublication: MavenPublication,
103103
connection.set("scm:git:$codeRepo")
104104
developerConnection.set("scm:git:$codeRepo")
105105
url.set("$codeRepo/tree/main")
106-
tag.set("main")
106+
val version = project.version.toString()
107+
if (!version.endsWith("-SNAPSHOT")) {
108+
tag.set("apache-polaris-$version")
109+
}
107110
}
108111
issueManagement { url.set(projectPeople.bugDatabase) }
109112

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one
3+
* or more contributor license agreements. See the NOTICE file
4+
* distributed with this work for additional information
5+
* regarding copyright ownership. The ASF licenses this file
6+
* to you under the Apache License, Version 2.0 (the
7+
* "License"); you may not use this file except in compliance
8+
* with the License. You may obtain a copy of the License at
9+
*
10+
* http://www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing,
13+
* software distributed under the License is distributed on an
14+
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
* KIND, either express or implied. See the License for the
16+
* specific language governing permissions and limitations
17+
* under the License.
18+
*/
19+
20+
package publishing
21+
22+
import java.security.MessageDigest
23+
import javax.inject.Inject
24+
import org.gradle.api.DefaultTask
25+
import org.gradle.api.model.ObjectFactory
26+
import org.gradle.api.tasks.Input
27+
import org.gradle.api.tasks.InputFile
28+
import org.gradle.api.tasks.OutputFile
29+
import org.gradle.api.tasks.TaskAction
30+
import org.gradle.work.DisableCachingByDefault
31+
32+
@DisableCachingByDefault
33+
abstract class GenerateDigest @Inject constructor(objectFactory: ObjectFactory) : DefaultTask() {
34+
35+
@get:InputFile val file = objectFactory.fileProperty()
36+
@get:Input val algorithm = objectFactory.property(String::class.java).convention("SHA-512")
37+
@get:OutputFile
38+
val outputFile =
39+
objectFactory.fileProperty().convention {
40+
val input = file.get().asFile
41+
val algo = algorithm.get()
42+
input.parentFile.resolve("${input.name}-${algo.replace("-", "").lowercase()}")
43+
}
44+
45+
@TaskAction
46+
fun generate() {
47+
val input = file.get().asFile
48+
val digestFile = outputFile.get().asFile
49+
val md = MessageDigest.getInstance(algorithm.get())
50+
input.inputStream().use {
51+
val buffered = it.buffered(8192)
52+
val buf = ByteArray(8192)
53+
var rd: Int
54+
while (true) {
55+
rd = buffered.read(buf)
56+
if (rd == -1) break
57+
md.update(buf, 0, rd)
58+
}
59+
60+
digestFile.writeText(
61+
md.digest().joinToString(separator = "") { eachByte -> "%02x".format(eachByte) } +
62+
" ${input.name}"
63+
)
64+
}
65+
}
66+
}

build-logic/src/main/kotlin/publishing/rootProject.kt

Lines changed: 16 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -61,27 +61,28 @@ internal fun configureOnRootProject(project: Project) =
6161
workingDir(project.projectDir)
6262
}
6363

64-
val digestSourceTarball = tasks.register("digestSourceTarball")
65-
digestSourceTarball.configure {
66-
mustRunAfter(sourceTarball)
67-
68-
doFirst {
69-
val e = project.extensions.getByType(PublishingHelperExtension::class.java)
70-
generateDigest(e.sourceTarball.get().asFile, e.sourceTarballDigest.get().asFile, "SHA-512")
64+
val digestSourceTarball =
65+
tasks.register<GenerateDigest>("digestSourceTarball") {
66+
description = "Generate the source tarball digest"
67+
mustRunAfter(sourceTarball)
68+
file.set {
69+
val e = project.extensions.getByType(PublishingHelperExtension::class.java)
70+
e.sourceTarball.get().asFile
71+
}
7172
}
72-
}
7373

7474
sourceTarball.configure { finalizedBy(digestSourceTarball) }
7575

7676
if (isSigning) {
77-
val signSourceTarball = tasks.register<Sign>("signSourceTarball")
78-
signSourceTarball.configure {
79-
mustRunAfter(sourceTarball)
80-
doFirst {
81-
val e = project.extensions.getByType(PublishingHelperExtension::class.java)
82-
sign(e.sourceTarball.get().asFile)
77+
val signSourceTarball =
78+
tasks.register<Sign>("signSourceTarball") {
79+
description = "Sign the source tarball"
80+
mustRunAfter(sourceTarball)
81+
doFirst {
82+
val e = project.extensions.getByType(PublishingHelperExtension::class.java)
83+
sign(e.sourceTarball.get().asFile)
84+
}
8385
}
84-
}
8586
sourceTarball.configure { finalizedBy(signSourceTarball) }
8687
}
8788

build-logic/src/main/kotlin/publishing/util.kt

Lines changed: 0 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -23,10 +23,8 @@ import groovy.json.JsonException
2323
import groovy.json.JsonSlurper
2424
import groovy.util.Node
2525
import groovy.util.NodeList
26-
import java.io.File
2726
import java.io.FileNotFoundException
2827
import java.net.URI
29-
import java.security.MessageDigest
3028
import org.gradle.api.artifacts.Configuration
3129
import org.gradle.api.artifacts.component.ModuleComponentSelector
3230
import org.gradle.api.artifacts.result.DependencyResult
@@ -58,25 +56,6 @@ internal fun xmlNode(node: Node?, child: String): Node? {
5856
return null
5957
}
6058

61-
internal fun generateDigest(input: File, output: File, algorithm: String) {
62-
val md = MessageDigest.getInstance(algorithm)
63-
input.inputStream().use {
64-
val buffered = it.buffered(8192)
65-
val buf = ByteArray(8192)
66-
var rd: Int
67-
while (true) {
68-
rd = buffered.read(buf)
69-
if (rd == -1) break
70-
md.update(buf, 0, rd)
71-
}
72-
73-
output.writeText(
74-
md.digest().joinToString(separator = "") { eachByte -> "%02x".format(eachByte) } +
75-
" ${input.name}"
76-
)
77-
}
78-
}
79-
8059
internal fun <T : Any> unsafeCast(o: Any?): T {
8160
@Suppress("UNCHECKED_CAST")
8261
return o as T

client/python/pyproject.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ urllib3 = "^1.25.3"
3636
python-dateutil = ">=2.8.2"
3737
pydantic = ">=2"
3838
typing-extensions = ">=4.7.1"
39-
boto3 = "==1.37.23"
39+
boto3 = "==1.37.28"
4040

4141
[tool.poetry.dev-dependencies]
4242
pytest = ">=7.2.1"

0 commit comments

Comments
 (0)